* [PATCH v3 0/1] Add PCD to disable safe string constraint assertions @ 2020-01-03 17:12 Vitaly Cheptsov 2020-01-03 17:12 ` [PATCH v3 1/1] MdePkg: " Vitaly Cheptsov 2020-01-06 18:28 ` [edk2-devel] [PATCH v3 0/1] " Michael D Kinney 0 siblings, 2 replies; 28+ messages in thread From: Vitaly Cheptsov @ 2020-01-03 17:12 UTC (permalink / raw) To: devel [-- Attachment #1: Type: text/plain, Size: 687 bytes --] REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2054 Requesting for merge in edk2-stable202002. Changes since V1: - Enable assertions by default to preserve the original behaviour - Fix bugzilla reference link - Update documentation in BaseLib.h Vitaly Cheptsov (1): MdePkg: Add PCD to disable safe string constraint assertions MdePkg/MdePkg.dec | 6 ++ MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- MdePkg/Include/Library/BaseLib.h | 74 +++++++++++++------- MdePkg/Library/BaseLib/SafeString.c | 4 +- MdePkg/MdePkg.uni | 6 ++ 5 files changed, 71 insertions(+), 30 deletions(-) -- 2.21.0 (Apple Git-122.2) [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 489 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH v3 1/1] MdePkg: Add PCD to disable safe string constraint assertions 2020-01-03 17:12 [PATCH v3 0/1] Add PCD to disable safe string constraint assertions Vitaly Cheptsov @ 2020-01-03 17:12 ` Vitaly Cheptsov 2020-01-06 18:28 ` [edk2-devel] [PATCH v3 0/1] " Michael D Kinney 1 sibling, 0 replies; 28+ messages in thread From: Vitaly Cheptsov @ 2020-01-03 17:12 UTC (permalink / raw) To: devel [-- Attachment #1: Type: text/plain, Size: 17597 bytes --] REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2054 Runtime data checks are not meant to cause debug assertions unless explicitly needed by some debug code (thus the PCD) as this breaks debug builds validating data with BaseLib. Signed-off-by: Vitaly Cheptsov <vit9696@protonmail.com> --- MdePkg/MdePkg.dec | 6 ++ MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- MdePkg/Include/Library/BaseLib.h | 74 +++++++++++++------- MdePkg/Library/BaseLib/SafeString.c | 4 +- MdePkg/MdePkg.uni | 6 ++ 5 files changed, 71 insertions(+), 30 deletions(-) diff --git a/MdePkg/MdePkg.dec b/MdePkg/MdePkg.dec index d022cc5e3e..0191b7a08b 100644 --- a/MdePkg/MdePkg.dec +++ b/MdePkg/MdePkg.dec @@ -2221,6 +2221,12 @@ [PcdsFixedAtBuild,PcdsPatchableInModule] # @Prompt Memory Address of GuidedExtractHandler Table. gEfiMdePkgTokenSpaceGuid.PcdGuidedExtractHandlerTableAddress|0x1000000|UINT64|0x30001015 + ## Indicates if safe string constraint violation should assert.<BR><BR> + # TRUE - Safe string constraint violation causes assertion.<BR> + # FALSE - Safe string constraint violation does not cause assertion.<BR> + # @Prompt Enable safe string constraint violation assertions. + gEfiMdePkgTokenSpaceGuid.PcdAssertOnSafeStringConstraints|TRUE|BOOLEAN|0x0000002e + [PcdsFixedAtBuild, PcdsPatchableInModule, PcdsDynamic, PcdsDynamicEx] ## This value is used to set the base address of PCI express hierarchy. # @Prompt PCI Express Base Address. diff --git a/MdePkg/Library/BaseLib/BaseLib.inf b/MdePkg/Library/BaseLib/BaseLib.inf index 3586beb0ab..bc98bc6134 100644 --- a/MdePkg/Library/BaseLib/BaseLib.inf +++ b/MdePkg/Library/BaseLib/BaseLib.inf @@ -390,11 +390,12 @@ [LibraryClasses] BaseMemoryLib [Pcd] - gEfiMdePkgTokenSpaceGuid.PcdMaximumLinkedListLength ## SOMETIMES_CONSUMES - gEfiMdePkgTokenSpaceGuid.PcdMaximumAsciiStringLength ## SOMETIMES_CONSUMES - gEfiMdePkgTokenSpaceGuid.PcdMaximumUnicodeStringLength ## SOMETIMES_CONSUMES - gEfiMdePkgTokenSpaceGuid.PcdControlFlowEnforcementPropertyMask ## SOMETIMES_CONSUMES - gEfiMdePkgTokenSpaceGuid.PcdSpeculationBarrierType ## SOMETIMES_CONSUMES + gEfiMdePkgTokenSpaceGuid.PcdAssertOnSafeStringConstraints ## SOMETIMES_CONSUMES + gEfiMdePkgTokenSpaceGuid.PcdMaximumLinkedListLength ## SOMETIMES_CONSUMES + gEfiMdePkgTokenSpaceGuid.PcdMaximumAsciiStringLength ## SOMETIMES_CONSUMES + gEfiMdePkgTokenSpaceGuid.PcdMaximumUnicodeStringLength ## SOMETIMES_CONSUMES + gEfiMdePkgTokenSpaceGuid.PcdControlFlowEnforcementPropertyMask ## SOMETIMES_CONSUMES + gEfiMdePkgTokenSpaceGuid.PcdSpeculationBarrierType ## SOMETIMES_CONSUMES [FeaturePcd] gEfiMdePkgTokenSpaceGuid.PcdVerifyNodeInList ## CONSUMES diff --git a/MdePkg/Include/Library/BaseLib.h b/MdePkg/Include/Library/BaseLib.h index 2a75bc023f..c413ca5f57 100644 --- a/MdePkg/Include/Library/BaseLib.h +++ b/MdePkg/Include/Library/BaseLib.h @@ -189,6 +189,8 @@ StrnSizeS ( If Destination is not aligned on a 16-bit boundary, then ASSERT(). If Source is not aligned on a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If an error would be returned, then the function will also ASSERT(). If an error is returned, then the Destination is unmodified. @@ -225,6 +227,8 @@ StrCpyS ( If Length > 0 and Destination is not aligned on a 16-bit boundary, then ASSERT(). If Length > 0 and Source is not aligned on a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If an error would be returned, then the function will also ASSERT(). If an error is returned, then the Destination is unmodified. @@ -263,6 +267,8 @@ StrnCpyS ( If Destination is not aligned on a 16-bit boundary, then ASSERT(). If Source is not aligned on a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If an error would be returned, then the function will also ASSERT(). If an error is returned, then the Destination is unmodified. @@ -303,6 +309,8 @@ StrCatS ( If Destination is not aligned on a 16-bit boundary, then ASSERT(). If Source is not aligned on a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If an error would be returned, then the function will also ASSERT(). If an error is returned, then the Destination is unmodified. @@ -350,9 +358,11 @@ StrnCatS ( be ignored. Then, the function stops at the first character that is a not a valid decimal character or a Null-terminator, whichever one comes first. + If String is not aligned in a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). If Data is NULL, then ASSERT(). - If String is not aligned in a 16-bit boundary, then ASSERT(). If PcdMaximumUnicodeStringLength is not zero, and String contains more than PcdMaximumUnicodeStringLength Unicode characters, not including the Null-terminator, then ASSERT(). @@ -406,9 +416,11 @@ StrDecimalToUintnS ( be ignored. Then, the function stops at the first character that is a not a valid decimal character or a Null-terminator, whichever one comes first. + If String is not aligned in a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). If Data is NULL, then ASSERT(). - If String is not aligned in a 16-bit boundary, then ASSERT(). If PcdMaximumUnicodeStringLength is not zero, and String contains more than PcdMaximumUnicodeStringLength Unicode characters, not including the Null-terminator, then ASSERT(). @@ -467,9 +479,11 @@ StrDecimalToUint64S ( the first character that is a not a valid hexadecimal character or NULL, whichever one comes first. + If String is not aligned in a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). If Data is NULL, then ASSERT(). - If String is not aligned in a 16-bit boundary, then ASSERT(). If PcdMaximumUnicodeStringLength is not zero, and String contains more than PcdMaximumUnicodeStringLength Unicode characters, not including the Null-terminator, then ASSERT(). @@ -528,9 +542,11 @@ StrHexToUintnS ( the first character that is a not a valid hexadecimal character or NULL, whichever one comes first. + If String is not aligned in a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). If Data is NULL, then ASSERT(). - If String is not aligned in a 16-bit boundary, then ASSERT(). If PcdMaximumUnicodeStringLength is not zero, and String contains more than PcdMaximumUnicodeStringLength Unicode characters, not including the Null-terminator, then ASSERT(). @@ -622,6 +638,7 @@ AsciiStrnSizeS ( This function is similar as strcpy_s defined in C11. + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If an error would be returned, then the function will also ASSERT(). If an error is returned, then the Destination is unmodified. @@ -656,6 +673,7 @@ AsciiStrCpyS ( This function is similar as strncpy_s defined in C11. + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If an error would be returned, then the function will also ASSERT(). If an error is returned, then the Destination is unmodified. @@ -692,6 +710,7 @@ AsciiStrnCpyS ( This function is similar as strcat_s defined in C11. + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If an error would be returned, then the function will also ASSERT(). If an error is returned, then the Destination is unmodified. @@ -730,6 +749,7 @@ AsciiStrCatS ( This function is similar as strncat_s defined in C11. + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If an error would be returned, then the function will also ASSERT(). If an error is returned, then the Destination is unmodified. @@ -777,6 +797,7 @@ AsciiStrnCatS ( be ignored. Then, the function stops at the first character that is a not a valid decimal character or a Null-terminator, whichever one comes first. + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). If Data is NULL, then ASSERT(). If PcdMaximumAsciiStringLength is not zero, and String contains more than @@ -832,6 +853,7 @@ AsciiStrDecimalToUintnS ( be ignored. Then, the function stops at the first character that is a not a valid decimal character or a Null-terminator, whichever one comes first. + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). If Data is NULL, then ASSERT(). If PcdMaximumAsciiStringLength is not zero, and String contains more than @@ -891,6 +913,7 @@ AsciiStrDecimalToUint64S ( character that is a not a valid hexadecimal character or Null-terminator, whichever on comes first. + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). If Data is NULL, then ASSERT(). If PcdMaximumAsciiStringLength is not zero, and String contains more than @@ -950,6 +973,7 @@ AsciiStrHexToUintnS ( character that is a not a valid hexadecimal character or Null-terminator, whichever on comes first. + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). If Data is NULL, then ASSERT(). If PcdMaximumAsciiStringLength is not zero, and String contains more than @@ -1506,12 +1530,11 @@ StrHexToUint64 ( "::" can be used to compress one or more groups of X when X contains only 0. The "::" can only appear once in the String. + If String is not aligned in a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). - If Address is NULL, then ASSERT(). - - If String is not aligned in a 16-bit boundary, then ASSERT(). - If PcdMaximumUnicodeStringLength is not zero, and String contains more than PcdMaximumUnicodeStringLength Unicode characters, not including the Null-terminator, then ASSERT(). @@ -1567,12 +1590,11 @@ StrToIpv6Address ( When /P is in the String, the function stops at the first character that is not a valid decimal digit character after P is converted. + If String is not aligned in a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). - If Address is NULL, then ASSERT(). - - If String is not aligned in a 16-bit boundary, then ASSERT(). - If PcdMaximumUnicodeStringLength is not zero, and String contains more than PcdMaximumUnicodeStringLength Unicode characters, not including the Null-terminator, then ASSERT(). @@ -1640,9 +1662,11 @@ StrToIpv4Address ( oo Data4[48:55] pp Data4[56:63] + If String is not aligned in a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). If Guid is NULL, then ASSERT(). - If String is not aligned in a 16-bit boundary, then ASSERT(). @param String Pointer to a Null-terminated Unicode string. @param Guid Pointer to the converted GUID. @@ -1676,15 +1700,12 @@ StrToGuid ( If String is not aligned in a 16-bit boundary, then ASSERT(). + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). - If Buffer is NULL, then ASSERT(). - If Length is not multiple of 2, then ASSERT(). - If PcdMaximumUnicodeStringLength is not zero and Length is greater than PcdMaximumUnicodeStringLength, then ASSERT(). - If MaxBufferSize is less than (Length / 2), then ASSERT(). @param String Pointer to a Null-terminated Unicode string. @@ -1775,8 +1796,9 @@ UnicodeStrToAsciiStr ( If any Unicode characters in Source contain non-zero value in the upper 8 bits, then ASSERT(). - If Source is not aligned on a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If an error would be returned, then the function will also ASSERT(). If an error is returned, then the Destination is unmodified. @@ -1824,6 +1846,8 @@ UnicodeStrToAsciiStrS ( If any Unicode characters in Source contain non-zero value in the upper 8 bits, then ASSERT(). If Source is not aligned on a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If an error would be returned, then the function will also ASSERT(). If an error is returned, then the Destination is unmodified. @@ -2388,8 +2412,8 @@ AsciiStrHexToUint64 ( "::" can be used to compress one or more groups of X when X contains only 0. The "::" can only appear once in the String. + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). - If Address is NULL, then ASSERT(). If EndPointer is not NULL and Address is translated from String, a pointer @@ -2443,8 +2467,8 @@ AsciiStrToIpv6Address ( When /P is in the String, the function stops at the first character that is not a valid decimal digit character after P is converted. + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). - If Address is NULL, then ASSERT(). If EndPointer is not NULL and Address is translated from String, a pointer @@ -2508,6 +2532,7 @@ AsciiStrToIpv4Address ( oo Data4[48:55] pp Data4[56:63] + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). If Guid is NULL, then ASSERT(). @@ -2541,15 +2566,12 @@ AsciiStrToGuid ( decoding stops after Length of characters and outputs Buffer containing (Length / 2) bytes. + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If String is NULL, then ASSERT(). - If Buffer is NULL, then ASSERT(). - If Length is not multiple of 2, then ASSERT(). - If PcdMaximumAsciiStringLength is not zero and Length is greater than PcdMaximumAsciiStringLength, then ASSERT(). - If MaxBufferSize is less than (Length / 2), then ASSERT(). @param String Pointer to a Null-terminated ASCII string. @@ -2632,6 +2654,8 @@ AsciiStrToUnicodeStr ( equal or greater than ((AsciiStrLen (Source) + 1) * sizeof (CHAR16)) in bytes. If Destination is not aligned on a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If an error would be returned, then the function will also ASSERT(). If an error is returned, then the Destination is unmodified. @@ -2678,6 +2702,8 @@ AsciiStrToUnicodeStrS ( ((MIN(AsciiStrLen(Source), Length) + 1) * sizeof (CHAR8)) in bytes. If Destination is not aligned on a 16-bit boundary, then ASSERT(). + + Unless PcdAssertOnSafeStringConstraints is set to FALSE: If an error would be returned, then the function will also ASSERT(). If an error is returned, then Destination and DestinationLength are diff --git a/MdePkg/Library/BaseLib/SafeString.c b/MdePkg/Library/BaseLib/SafeString.c index 7dc03d2caa..56b5e34a8d 100644 --- a/MdePkg/Library/BaseLib/SafeString.c +++ b/MdePkg/Library/BaseLib/SafeString.c @@ -14,7 +14,9 @@ #define SAFE_STRING_CONSTRAINT_CHECK(Expression, Status) \ do { \ - ASSERT (Expression); \ + if (PcdGetBool (PcdAssertOnSafeStringConstraints)) { \ + ASSERT (Expression); \ + } \ if (!(Expression)) { \ return Status; \ } \ diff --git a/MdePkg/MdePkg.uni b/MdePkg/MdePkg.uni index 5c1fa24065..425b66bb43 100644 --- a/MdePkg/MdePkg.uni +++ b/MdePkg/MdePkg.uni @@ -287,6 +287,12 @@ #string STR_gEfiMdePkgTokenSpaceGuid_PcdGuidedExtractHandlerTableAddress_HELP #language en-US "This value is used to set the available memory address to store Guided Extract Handlers. The required memory space is decided by the value of PcdMaximumGuidedExtractHandler." +#string STR_gEfiMdePkgTokenSpaceGuid_PcdAssertOnSafeStringConstraints_PROMPT #language en-US "Enable safe string constraint violation assertions" + +#string STR_gEfiMdePkgTokenSpaceGuid_PcdAssertOnSafeStringConstraints_HELP #language en-US "Indicates if safe string constraint violation should assert.<BR><BR>\n" + "TRUE - Safe string constraint violation causes assertion.<BR>\n" + "FALSE - Safe string constraint violation does not cause assertion.<BR>" + #string STR_gEfiMdePkgTokenSpaceGuid_PcdPciExpressBaseAddress_PROMPT #language en-US "PCI Express Base Address" #string STR_gEfiMdePkgTokenSpaceGuid_PcdPciExpressBaseAddress_HELP #language en-US "This value is used to set the base address of PCI express hierarchy." -- 2.21.0 (Apple Git-122.2) [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 489 bytes --] ^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-01-03 17:12 [PATCH v3 0/1] Add PCD to disable safe string constraint assertions Vitaly Cheptsov 2020-01-03 17:12 ` [PATCH v3 1/1] MdePkg: " Vitaly Cheptsov @ 2020-01-06 18:28 ` Michael D Kinney 2020-01-06 18:43 ` Vitaly Cheptsov 1 sibling, 1 reply; 28+ messages in thread From: Michael D Kinney @ 2020-01-06 18:28 UTC (permalink / raw) To: devel@edk2.groups.io, vit9696@protonmail.com, Kinney, Michael D Hi Vitaly, Is the use case for UEFI Applications? There is a different mechanism to disable all ASSERT() statements within a UEFI Application. If a component is consuming data from an untrusted source, then that component is required to verify the untrusted data before passing it to a function that clearly documents is input requirements. If this approach is followed, then the BaseLib functions can be used "as is" as long as the ASSERT() conditions are verified before calling. If there are some APIs that currently document their ASSERT() behavior and we think that ASSERT() behavior is incorrect and should be handled by an existing error return value, then we should discuss each of those APIs individually. Mike > -----Original Message----- > From: devel@edk2.groups.io <devel@edk2.groups.io> On > Behalf Of Vitaly Cheptsov via Groups.Io > Sent: Friday, January 3, 2020 9:13 AM > To: devel@edk2.groups.io > Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to disable > safe string constraint assertions > > REF: > https://bugzilla.tianocore.org/show_bug.cgi?id=2054 > > Requesting for merge in edk2-stable202002. > > Changes since V1: > - Enable assertions by default to preserve the original > behaviour > - Fix bugzilla reference link > - Update documentation in BaseLib.h > > Vitaly Cheptsov (1): > MdePkg: Add PCD to disable safe string constraint > assertions > > MdePkg/MdePkg.dec | 6 ++ > MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- > MdePkg/Include/Library/BaseLib.h | 74 > +++++++++++++------- > MdePkg/Library/BaseLib/SafeString.c | 4 +- > MdePkg/MdePkg.uni | 6 ++ > 5 files changed, 71 insertions(+), 30 deletions(-) > > -- > 2.21.0 (Apple Git-122.2) > > > -=-=-=-=-=-= > Groups.io Links: You receive all messages sent to this > group. > > View/Reply Online (#52837): > https://edk2.groups.io/g/devel/message/52837 > Mute This Topic: https://groups.io/mt/69401948/1643496 > Group Owner: devel+owner@edk2.groups.io > Unsubscribe: https://edk2.groups.io/g/devel/unsub > [michael.d.kinney@intel.com] > -=-=-=-=-=-= ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-01-06 18:28 ` [edk2-devel] [PATCH v3 0/1] " Michael D Kinney @ 2020-01-06 18:43 ` Vitaly Cheptsov 2020-01-06 22:54 ` Sean 2020-01-08 16:35 ` Michael D Kinney 0 siblings, 2 replies; 28+ messages in thread From: Vitaly Cheptsov @ 2020-01-06 18:43 UTC (permalink / raw) To: Kinney, Michael D; +Cc: devel@edk2.groups.io [-- Attachment #1: Type: text/plain, Size: 3779 bytes --] Hi Mike, Yes, the primary use case is for UEFI Applications. We do not want to disable ASSERT’s completely, as assertions that make sense, i.e. the ones signalising about interface misuse, are helpful for debugging. I have already explained in the BZ that basically all safe string constraint assertions make no sense for handling untrusted data. We find this use case very logical, as these functions behave properly with assertions disabled and cover all these error conditions by the return statuses. In such situation is not useful for these functions to assert, as we end up inefficiently reimplementing the logic. I would have liked the approach of discussing the interfaces individually, but I struggle to find any that makes sense from this point of view. AsciiStrToGuid will ASSERT when the length of the passed string is odd. Functions that cannot, ahem, parse, for us are pretty much useless. AsciiStrCatS will ASSERT when the appended string does not fit the buffer. For us this logic makes this function pretty much equivalent to deprecated and thus unavailable AsciiStrCat, except it is also slower. My original suggestion was to remove the assertions entirely, but several people here said that they use them to verify usage errors when handling trusted data. This makes good sense to me, so we suggest to support both cases by introducing a PCD in this patch. Best wishes, Vitaly > 6 янв. 2020 г., в 21:28, Kinney, Michael D <michael.d.kinney@intel.com> написал(а): > > > Hi Vitaly, > > Is the use case for UEFI Applications? > > There is a different mechanism to disable all ASSERT() > statements within a UEFI Application. > > If a component is consuming data from an untrusted source, > then that component is required to verify the untrusted > data before passing it to a function that clearly documents > is input requirements. If this approach is followed, then > the BaseLib functions can be used "as is" as long as the > ASSERT() conditions are verified before calling. > > If there are some APIs that currently document their ASSERT() > behavior and we think that ASSERT() behavior is incorrect and > should be handled by an existing error return value, then we > should discuss each of those APIs individually. > > Mike > > >> -----Original Message----- >> From: devel@edk2.groups.io <devel@edk2.groups.io> On >> Behalf Of Vitaly Cheptsov via Groups.Io >> Sent: Friday, January 3, 2020 9:13 AM >> To: devel@edk2.groups.io >> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to disable >> safe string constraint assertions >> >> REF: >> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >> >> Requesting for merge in edk2-stable202002. >> >> Changes since V1: >> - Enable assertions by default to preserve the original >> behaviour >> - Fix bugzilla reference link >> - Update documentation in BaseLib.h >> >> Vitaly Cheptsov (1): >> MdePkg: Add PCD to disable safe string constraint >> assertions >> >> MdePkg/MdePkg.dec | 6 ++ >> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >> MdePkg/Include/Library/BaseLib.h | 74 >> +++++++++++++------- >> MdePkg/Library/BaseLib/SafeString.c | 4 +- >> MdePkg/MdePkg.uni | 6 ++ >> 5 files changed, 71 insertions(+), 30 deletions(-) >> >> -- >> 2.21.0 (Apple Git-122.2) >> >> >> -=-=-=-=-=-= >> Groups.io Links: You receive all messages sent to this >> group. >> >> View/Reply Online (#52837): >> https://edk2.groups.io/g/devel/message/52837 >> Mute This Topic: https://groups.io/mt/69401948/1643496 >> Group Owner: devel+owner@edk2.groups.io >> Unsubscribe: https://edk2.groups.io/g/devel/unsub >> [michael.d.kinney@intel.com] >> -=-=-=-=-=-= > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 489 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-01-06 18:43 ` Vitaly Cheptsov @ 2020-01-06 22:54 ` Sean 2020-01-08 16:35 ` Michael D Kinney 1 sibling, 0 replies; 28+ messages in thread From: Sean @ 2020-01-06 22:54 UTC (permalink / raw) To: vit9696, devel [-- Attachment #1: Type: text/plain, Size: 913 bytes --] On Mon, Jan 6, 2020 at 10:43 AM, Vitaly Cheptsov wrote: > > My original suggestion was to remove the assertions entirely, but several > people here said that they use them to verify usage errors when handling > trusted data. This makes good sense to me, so we suggest to support both > cases by introducing a PCD in this patch. I strongly agree with Vitaly. These asserts cause more trouble than help. The asserts cause the caller to implement the same checks as the functions and thus make the functional checks useless overhead and complicate the calling code. These functions are in a base library used in hundreds of unique places and thus should not make assumptions about how to handle errors. Since they have the ability to and must return error codes (since asserts are generally off in production) this code should rely on the caller to handle the error appropriately. thanks Sean [-- Attachment #2: Type: text/html, Size: 977 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-01-06 18:43 ` Vitaly Cheptsov 2020-01-06 22:54 ` Sean @ 2020-01-08 16:35 ` Michael D Kinney 2020-01-27 9:47 ` Vitaly Cheptsov 1 sibling, 1 reply; 28+ messages in thread From: Michael D Kinney @ 2020-01-08 16:35 UTC (permalink / raw) To: devel@edk2.groups.io, vit9696@protonmail.com, Kinney, Michael D Hi Vitaly, Thanks for the additional background. I would like a couple extra day to review the PCD name and the places the PCD might potentially be used. If we find other APIs where ASSERT() behavior is only valuable during dev/debug to quickly identify misuse with trusted data and the API provides predicable return behavior when ASSERT() is disabled, then I would like to have a pattern we can potentially apply to all these APIs across all packages. Thanks, Mike > -----Original Message----- > From: devel@edk2.groups.io <devel@edk2.groups.io> On > Behalf Of Vitaly Cheptsov via Groups.Io > Sent: Monday, January 6, 2020 10:44 AM > To: Kinney, Michael D <michael.d.kinney@intel.com> > Cc: devel@edk2.groups.io > Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to > disable safe string constraint assertions > > Hi Mike, > > Yes, the primary use case is for UEFI Applications. We > do not want to disable ASSERT’s completely, as > assertions that make sense, i.e. the ones signalising > about interface misuse, are helpful for debugging. > > I have already explained in the BZ that basically all > safe string constraint assertions make no sense for > handling untrusted data. We find this use case very > logical, as these functions behave properly with > assertions disabled and cover all these error > conditions by the return statuses. In such situation is > not useful for these functions to assert, as we end up > inefficiently reimplementing the logic. I would have > liked the approach of discussing the interfaces > individually, but I struggle to find any that makes > sense from this point of view. > > AsciiStrToGuid will ASSERT when the length of the > passed string is odd. Functions that cannot, ahem, > parse, for us are pretty much useless. > AsciiStrCatS will ASSERT when the appended string does > not fit the buffer. For us this logic makes this > function pretty much equivalent to deprecated and thus > unavailable AsciiStrCat, except it is also slower. > > My original suggestion was to remove the assertions > entirely, but several people here said that they use > them to verify usage errors when handling trusted data. > This makes good sense to me, so we suggest to support > both cases by introducing a PCD in this patch. > > Best wishes, > Vitaly > > > 6 янв. 2020 г., в 21:28, Kinney, Michael D > <michael.d.kinney@intel.com> написал(а): > > > > > > Hi Vitaly, > > > > Is the use case for UEFI Applications? > > > > There is a different mechanism to disable all > ASSERT() > > statements within a UEFI Application. > > > > If a component is consuming data from an untrusted > source, > > then that component is required to verify the > untrusted > > data before passing it to a function that clearly > documents > > is input requirements. If this approach is followed, > then > > the BaseLib functions can be used "as is" as long as > the > > ASSERT() conditions are verified before calling. > > > > If there are some APIs that currently document their > ASSERT() > > behavior and we think that ASSERT() behavior is > incorrect and > > should be handled by an existing error return value, > then we > > should discuss each of those APIs individually. > > > > Mike > > > > > >> -----Original Message----- > >> From: devel@edk2.groups.io <devel@edk2.groups.io> On > >> Behalf Of Vitaly Cheptsov via Groups.Io > >> Sent: Friday, January 3, 2020 9:13 AM > >> To: devel@edk2.groups.io > >> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to > disable > >> safe string constraint assertions > >> > >> REF: > >> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 > >> > >> Requesting for merge in edk2-stable202002. > >> > >> Changes since V1: > >> - Enable assertions by default to preserve the > original > >> behaviour > >> - Fix bugzilla reference link > >> - Update documentation in BaseLib.h > >> > >> Vitaly Cheptsov (1): > >> MdePkg: Add PCD to disable safe string constraint > >> assertions > >> > >> MdePkg/MdePkg.dec | 6 ++ > >> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- > >> MdePkg/Include/Library/BaseLib.h | 74 > >> +++++++++++++------- > >> MdePkg/Library/BaseLib/SafeString.c | 4 +- > >> MdePkg/MdePkg.uni | 6 ++ > >> 5 files changed, 71 insertions(+), 30 deletions(-) > >> > >> -- > >> 2.21.0 (Apple Git-122.2) > >> > >> > >> -=-=-=-=-=-= > >> Groups.io Links: You receive all messages sent to > this > >> group. > >> > >> View/Reply Online (#52837): > >> https://edk2.groups.io/g/devel/message/52837 > >> Mute This Topic: > https://groups.io/mt/69401948/1643496 > >> Group Owner: devel+owner@edk2.groups.io > >> Unsubscribe: https://edk2.groups.io/g/devel/unsub > >> [michael.d.kinney@intel.com] > >> -=-=-=-=-=-= > > > > > ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-01-08 16:35 ` Michael D Kinney @ 2020-01-27 9:47 ` Vitaly Cheptsov 2020-02-10 11:12 ` Vitaly Cheptsov 0 siblings, 1 reply; 28+ messages in thread From: Vitaly Cheptsov @ 2020-01-27 9:47 UTC (permalink / raw) To: Kinney, Michael D; +Cc: devel@edk2.groups.io [-- Attachment #1: Type: text/plain, Size: 5251 bytes --] Hi Mike, Any progress with this? We would really benefit from this landing in the next stable release. Best, Vitaly > 8 янв. 2020 г., в 19:35, Kinney, Michael D <michael.d.kinney@intel.com> написал(а): > > > Hi Vitaly, > > Thanks for the additional background. I would like > a couple extra day to review the PCD name and the places > the PCD might potentially be used. > > If we find other APIs where ASSERT() behavior is only > valuable during dev/debug to quickly identify misuse > with trusted data and the API provides predicable > return behavior when ASSERT() is disabled, then I would > like to have a pattern we can potentially apply to all > these APIs across all packages. > > Thanks, > > Mike > >> -----Original Message----- >> From: devel@edk2.groups.io <devel@edk2.groups.io> On >> Behalf Of Vitaly Cheptsov via Groups.Io >> Sent: Monday, January 6, 2020 10:44 AM >> To: Kinney, Michael D <michael.d.kinney@intel.com> >> Cc: devel@edk2.groups.io >> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >> disable safe string constraint assertions >> >> Hi Mike, >> >> Yes, the primary use case is for UEFI Applications. We >> do not want to disable ASSERT’s completely, as >> assertions that make sense, i.e. the ones signalising >> about interface misuse, are helpful for debugging. >> >> I have already explained in the BZ that basically all >> safe string constraint assertions make no sense for >> handling untrusted data. We find this use case very >> logical, as these functions behave properly with >> assertions disabled and cover all these error >> conditions by the return statuses. In such situation is >> not useful for these functions to assert, as we end up >> inefficiently reimplementing the logic. I would have >> liked the approach of discussing the interfaces >> individually, but I struggle to find any that makes >> sense from this point of view. >> >> AsciiStrToGuid will ASSERT when the length of the >> passed string is odd. Functions that cannot, ahem, >> parse, for us are pretty much useless. >> AsciiStrCatS will ASSERT when the appended string does >> not fit the buffer. For us this logic makes this >> function pretty much equivalent to deprecated and thus >> unavailable AsciiStrCat, except it is also slower. >> >> My original suggestion was to remove the assertions >> entirely, but several people here said that they use >> them to verify usage errors when handling trusted data. >> This makes good sense to me, so we suggest to support >> both cases by introducing a PCD in this patch. >> >> Best wishes, >> Vitaly >> >>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >> <michael.d.kinney@intel.com> написал(а): >>> >>> >>> Hi Vitaly, >>> >>> Is the use case for UEFI Applications? >>> >>> There is a different mechanism to disable all >> ASSERT() >>> statements within a UEFI Application. >>> >>> If a component is consuming data from an untrusted >> source, >>> then that component is required to verify the >> untrusted >>> data before passing it to a function that clearly >> documents >>> is input requirements. If this approach is followed, >> then >>> the BaseLib functions can be used "as is" as long as >> the >>> ASSERT() conditions are verified before calling. >>> >>> If there are some APIs that currently document their >> ASSERT() >>> behavior and we think that ASSERT() behavior is >> incorrect and >>> should be handled by an existing error return value, >> then we >>> should discuss each of those APIs individually. >>> >>> Mike >>> >>> >>>> -----Original Message----- >>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On >>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>> Sent: Friday, January 3, 2020 9:13 AM >>>> To: devel@edk2.groups.io >>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >> disable >>>> safe string constraint assertions >>>> >>>> REF: >>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >>>> >>>> Requesting for merge in edk2-stable202002. >>>> >>>> Changes since V1: >>>> - Enable assertions by default to preserve the >> original >>>> behaviour >>>> - Fix bugzilla reference link >>>> - Update documentation in BaseLib.h >>>> >>>> Vitaly Cheptsov (1): >>>> MdePkg: Add PCD to disable safe string constraint >>>> assertions >>>> >>>> MdePkg/MdePkg.dec | 6 ++ >>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >>>> MdePkg/Include/Library/BaseLib.h | 74 >>>> +++++++++++++------- >>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >>>> MdePkg/MdePkg.uni | 6 ++ >>>> 5 files changed, 71 insertions(+), 30 deletions(-) >>>> >>>> -- >>>> 2.21.0 (Apple Git-122.2) >>>> >>>> >>>> -=-=-=-=-=-= >>>> Groups.io Links: You receive all messages sent to >> this >>>> group. >>>> >>>> View/Reply Online (#52837): >>>> https://edk2.groups.io/g/devel/message/52837 >>>> Mute This Topic: >> https://groups.io/mt/69401948/1643496 >>>> Group Owner: devel+owner@edk2.groups.io >>>> Unsubscribe: https://edk2.groups.io/g/devel/unsub >>>> [michael.d.kinney@intel.com] >>>> -=-=-=-=-=-= >>> >> >> >> > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 489 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-01-27 9:47 ` Vitaly Cheptsov @ 2020-02-10 11:12 ` Vitaly Cheptsov 2020-02-14 11:54 ` Vitaly Cheptsov 0 siblings, 1 reply; 28+ messages in thread From: Vitaly Cheptsov @ 2020-02-10 11:12 UTC (permalink / raw) To: Kinney, Michael D, Gao, Liming, Gao, Zhichao, devel@edk2.groups.io Cc: Marvin Häuser, Laszlo Ersek [-- Attachment #1: Type: text/plain, Size: 5914 bytes --] Hello, It has been quite some time since we submitted the patch with so far no negative response. As I mentioned previously, my team will strongly benefit from its landing in EDK II mainline. Since it does not add any regressions and can be viewed as a feature implementation for the rest of EDK II users, I request this to be merged upstream in edk2-stable202002. Best wishes, Vitaly > 27 янв. 2020 г., в 12:47, vit9696 <vit9696@protonmail.com> написал(а): > > > Hi Mike, > > Any progress with this? We would really benefit from this landing in the next stable release. > > Best, > Vitaly > >> 8 янв. 2020 г., в 19:35, Kinney, Michael D <michael.d.kinney@intel.com> написал(а): >> >> >> Hi Vitaly, >> >> Thanks for the additional background. I would like >> a couple extra day to review the PCD name and the places >> the PCD might potentially be used. >> >> If we find other APIs where ASSERT() behavior is only >> valuable during dev/debug to quickly identify misuse >> with trusted data and the API provides predicable >> return behavior when ASSERT() is disabled, then I would >> like to have a pattern we can potentially apply to all >> these APIs across all packages. >> >> Thanks, >> >> Mike >> >>> -----Original Message----- >>> From: devel@edk2.groups.io <devel@edk2.groups.io> On >>> Behalf Of Vitaly Cheptsov via Groups.Io >>> Sent: Monday, January 6, 2020 10:44 AM >>> To: Kinney, Michael D <michael.d.kinney@intel.com> >>> Cc: devel@edk2.groups.io >>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>> disable safe string constraint assertions >>> >>> Hi Mike, >>> >>> Yes, the primary use case is for UEFI Applications. We >>> do not want to disable ASSERT’s completely, as >>> assertions that make sense, i.e. the ones signalising >>> about interface misuse, are helpful for debugging. >>> >>> I have already explained in the BZ that basically all >>> safe string constraint assertions make no sense for >>> handling untrusted data. We find this use case very >>> logical, as these functions behave properly with >>> assertions disabled and cover all these error >>> conditions by the return statuses. In such situation is >>> not useful for these functions to assert, as we end up >>> inefficiently reimplementing the logic. I would have >>> liked the approach of discussing the interfaces >>> individually, but I struggle to find any that makes >>> sense from this point of view. >>> >>> AsciiStrToGuid will ASSERT when the length of the >>> passed string is odd. Functions that cannot, ahem, >>> parse, for us are pretty much useless. >>> AsciiStrCatS will ASSERT when the appended string does >>> not fit the buffer. For us this logic makes this >>> function pretty much equivalent to deprecated and thus >>> unavailable AsciiStrCat, except it is also slower. >>> >>> My original suggestion was to remove the assertions >>> entirely, but several people here said that they use >>> them to verify usage errors when handling trusted data. >>> This makes good sense to me, so we suggest to support >>> both cases by introducing a PCD in this patch. >>> >>> Best wishes, >>> Vitaly >>> >>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >>> <michael.d.kinney@intel.com> написал(а): >>>> >>>> >>>> Hi Vitaly, >>>> >>>> Is the use case for UEFI Applications? >>>> >>>> There is a different mechanism to disable all >>> ASSERT() >>>> statements within a UEFI Application. >>>> >>>> If a component is consuming data from an untrusted >>> source, >>>> then that component is required to verify the >>> untrusted >>>> data before passing it to a function that clearly >>> documents >>>> is input requirements. If this approach is followed, >>> then >>>> the BaseLib functions can be used "as is" as long as >>> the >>>> ASSERT() conditions are verified before calling. >>>> >>>> If there are some APIs that currently document their >>> ASSERT() >>>> behavior and we think that ASSERT() behavior is >>> incorrect and >>>> should be handled by an existing error return value, >>> then we >>>> should discuss each of those APIs individually. >>>> >>>> Mike >>>> >>>> >>>>> -----Original Message----- >>>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On >>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>> Sent: Friday, January 3, 2020 9:13 AM >>>>> To: devel@edk2.groups.io >>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >>> disable >>>>> safe string constraint assertions >>>>> >>>>> REF: >>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >>>>> >>>>> Requesting for merge in edk2-stable202002. >>>>> >>>>> Changes since V1: >>>>> - Enable assertions by default to preserve the >>> original >>>>> behaviour >>>>> - Fix bugzilla reference link >>>>> - Update documentation in BaseLib.h >>>>> >>>>> Vitaly Cheptsov (1): >>>>> MdePkg: Add PCD to disable safe string constraint >>>>> assertions >>>>> >>>>> MdePkg/MdePkg.dec | 6 ++ >>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >>>>> MdePkg/Include/Library/BaseLib.h | 74 >>>>> +++++++++++++------- >>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >>>>> MdePkg/MdePkg.uni | 6 ++ >>>>> 5 files changed, 71 insertions(+), 30 deletions(-) >>>>> >>>>> -- >>>>> 2.21.0 (Apple Git-122.2) >>>>> >>>>> >>>>> -=-=-=-=-=-= >>>>> Groups.io Links: You receive all messages sent to >>> this >>>>> group. >>>>> >>>>> View/Reply Online (#52837): >>>>> https://edk2.groups.io/g/devel/message/52837 >>>>> Mute This Topic: >>> https://groups.io/mt/69401948/1643496 >>>>> Group Owner: devel+owner@edk2.groups.io >>>>> Unsubscribe: https://edk2.groups.io/g/devel/unsub >>>>> [michael.d.kinney@intel.com] >>>>> -=-=-=-=-=-= >>>> >>> >>> >>> >> > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 489 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-10 11:12 ` Vitaly Cheptsov @ 2020-02-14 11:54 ` Vitaly Cheptsov 2020-02-14 17:00 ` Michael D Kinney 0 siblings, 1 reply; 28+ messages in thread From: Vitaly Cheptsov @ 2020-02-14 11:54 UTC (permalink / raw) To: Kinney, Michael D, Gao, Liming, Gao, Zhichao, devel@edk2.groups.io Cc: Marvin Häuser, Laszlo Ersek [-- Attachment #1: Type: text/plain, Size: 5979 bytes --] Replying as per Liming's request for this to be merged into edk2-stable202002. On Mon, Feb 10, 2020 at 14:12, vit9696 <vit9696@protonmail.com> wrote: > Hello, > > It has been quite some time since we submitted the patch with so far no negative response. As I mentioned previously, my team will strongly benefit from its landing in EDK II mainline. Since it does not add any regressions and can be viewed as a feature implementation for the rest of EDK II users, I request this to be merged upstream in edk2-stable202002. > > Best wishes, > Vitaly > >> 27 янв. 2020 г., в 12:47, vit9696 <vit9696@protonmail.com> написал(а): >> >> >> Hi Mike, >> >> Any progress with this? We would really benefit from this landing in the next stable release. >> >> Best, >> Vitaly >> >>> 8 янв. 2020 г., в 19:35, Kinney, Michael D <michael.d.kinney@intel.com> написал(а): >>> >>> >>> Hi Vitaly, >>> >>> Thanks for the additional background. I would like >>> a couple extra day to review the PCD name and the places >>> the PCD might potentially be used. >>> >>> If we find other APIs where ASSERT() behavior is only >>> valuable during dev/debug to quickly identify misuse >>> with trusted data and the API provides predicable >>> return behavior when ASSERT() is disabled, then I would >>> like to have a pattern we can potentially apply to all >>> these APIs across all packages. >>> >>> Thanks, >>> >>> Mike >>> >>>> -----Original Message----- >>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On >>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>> Sent: Monday, January 6, 2020 10:44 AM >>>> To: Kinney, Michael D <michael.d.kinney@intel.com> >>>> Cc: devel@edk2.groups.io >>>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>> disable safe string constraint assertions >>>> >>>> Hi Mike, >>>> >>>> Yes, the primary use case is for UEFI Applications. We >>>> do not want to disable ASSERT’s completely, as >>>> assertions that make sense, i.e. the ones signalising >>>> about interface misuse, are helpful for debugging. >>>> >>>> I have already explained in the BZ that basically all >>>> safe string constraint assertions make no sense for >>>> handling untrusted data. We find this use case very >>>> logical, as these functions behave properly with >>>> assertions disabled and cover all these error >>>> conditions by the return statuses. In such situation is >>>> not useful for these functions to assert, as we end up >>>> inefficiently reimplementing the logic. I would have >>>> liked the approach of discussing the interfaces >>>> individually, but I struggle to find any that makes >>>> sense from this point of view. >>>> >>>> AsciiStrToGuid will ASSERT when the length of the >>>> passed string is odd. Functions that cannot, ahem, >>>> parse, for us are pretty much useless. >>>> AsciiStrCatS will ASSERT when the appended string does >>>> not fit the buffer. For us this logic makes this >>>> function pretty much equivalent to deprecated and thus >>>> unavailable AsciiStrCat, except it is also slower. >>>> >>>> My original suggestion was to remove the assertions >>>> entirely, but several people here said that they use >>>> them to verify usage errors when handling trusted data. >>>> This makes good sense to me, so we suggest to support >>>> both cases by introducing a PCD in this patch. >>>> >>>> Best wishes, >>>> Vitaly >>>> >>>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >>>> <michael.d.kinney@intel.com> написал(а): >>>>> >>>>> >>>>> Hi Vitaly, >>>>> >>>>> Is the use case for UEFI Applications? >>>>> >>>>> There is a different mechanism to disable all >>>> ASSERT() >>>>> statements within a UEFI Application. >>>>> >>>>> If a component is consuming data from an untrusted >>>> source, >>>>> then that component is required to verify the >>>> untrusted >>>>> data before passing it to a function that clearly >>>> documents >>>>> is input requirements. If this approach is followed, >>>> then >>>>> the BaseLib functions can be used "as is" as long as >>>> the >>>>> ASSERT() conditions are verified before calling. >>>>> >>>>> If there are some APIs that currently document their >>>> ASSERT() >>>>> behavior and we think that ASSERT() behavior is >>>> incorrect and >>>>> should be handled by an existing error return value, >>>> then we >>>>> should discuss each of those APIs individually. >>>>> >>>>> Mike >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On >>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>> Sent: Friday, January 3, 2020 9:13 AM >>>>>> To: devel@edk2.groups.io >>>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>> disable >>>>>> safe string constraint assertions >>>>>> >>>>>> REF: >>>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >>>>>> >>>>>> Requesting for merge in edk2-stable202002. >>>>>> >>>>>> Changes since V1: >>>>>> - Enable assertions by default to preserve the >>>> original >>>>>> behaviour >>>>>> - Fix bugzilla reference link >>>>>> - Update documentation in BaseLib.h >>>>>> >>>>>> Vitaly Cheptsov (1): >>>>>> MdePkg: Add PCD to disable safe string constraint >>>>>> assertions >>>>>> >>>>>> MdePkg/MdePkg.dec | 6 ++ >>>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >>>>>> MdePkg/Include/Library/BaseLib.h | 74 >>>>>> +++++++++++++------- >>>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >>>>>> MdePkg/MdePkg.uni | 6 ++ >>>>>> 5 files changed, 71 insertions(+), 30 deletions(-) >>>>>> >>>>>> -- >>>>>> 2.21.0 (Apple Git-122.2) >>>>>> >>>>>> >>>>>> -=-=-=-=-=-= >>>>>> Groups.io Links: You receive all messages sent to >>>> this >>>>>> group. >>>>>> >>>>>> View/Reply Online (#52837): >>>>>> https://edk2.groups.io/g/devel/message/52837 >>>>>> Mute This Topic: >>>> https://groups.io/mt/69401948/1643496 >>>>>> Group Owner: devel+owner@edk2.groups.io >>>>>> Unsubscribe: https://edk2.groups.io/g/devel/unsub >>>>>> [michael.d.kinney@intel.com] >>>>>> -=-=-=-=-=-= >>>>> >>>> >>>> >>>> >>> >> [-- Attachment #2: Type: text/html, Size: 8648 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-14 11:54 ` Vitaly Cheptsov @ 2020-02-14 17:00 ` Michael D Kinney 2020-02-14 17:37 ` Vitaly Cheptsov 0 siblings, 1 reply; 28+ messages in thread From: Michael D Kinney @ 2020-02-14 17:00 UTC (permalink / raw) To: devel@edk2.groups.io, vit9696@protonmail.com, Gao, Liming, Gao, Zhichao, Kinney, Michael D Cc: Marvin Häuser, Laszlo Ersek [-- Attachment #1: Type: text/plain, Size: 7282 bytes --] Vitaly, I want to make sure a feature PCD can be used to disable ASSERT() behavior in more than just safe string functions in BaseLib. Can we consider changing the name and description of PcdAssertOnSafeStringConstraints to be more generic, so if we find other lib APIs, the name will make sense? Maybe something like: PcdEnableLibraryAssertChecks? Default is TRUE. Can set to FALSE in DSC file to disable ASSERT() checks. Thanks, Mike From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Vitaly Cheptsov via Groups.Io Sent: Friday, February 14, 2020 3:55 AM To: Kinney, Michael D <michael.d.kinney@intel.com>; Gao, Liming <liming.gao@intel.com>; Gao, Zhichao <zhichao.gao@intel.com>; devel@edk2.groups.io Cc: Marvin Häuser <marvin.haeuser@outlook.com>; Laszlo Ersek <lersek@redhat.com> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions Replying as per Liming's request for this to be merged into edk2-stable202002. On Mon, Feb 10, 2020 at 14:12, vit9696 <vit9696@protonmail.com<mailto:vit9696@protonmail.com>> wrote: Hello, It has been quite some time since we submitted the patch with so far no negative response. As I mentioned previously, my team will strongly benefit from its landing in EDK II mainline. Since it does not add any regressions and can be viewed as a feature implementation for the rest of EDK II users, I request this to be merged upstream in edk2-stable202002. Best wishes, Vitaly > 27 янв. 2020 г., в 12:47, vit9696 <vit9696@protonmail.com<mailto:vit9696@protonmail.com>> написал(а): > > > Hi Mike, > > Any progress with this? We would really benefit from this landing in the next stable release. > > Best, > Vitaly > >> 8 янв. 2020 г., в 19:35, Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> написал(а): >> >> >> Hi Vitaly, >> >> Thanks for the additional background. I would like >> a couple extra day to review the PCD name and the places >> the PCD might potentially be used. >> >> If we find other APIs where ASSERT() behavior is only >> valuable during dev/debug to quickly identify misuse >> with trusted data and the API provides predicable >> return behavior when ASSERT() is disabled, then I would >> like to have a pattern we can potentially apply to all >> these APIs across all packages. >> >> Thanks, >> >> Mike >> >>> -----Original Message----- >>> From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On >>> Behalf Of Vitaly Cheptsov via Groups.Io >>> Sent: Monday, January 6, 2020 10:44 AM >>> To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> >>> Cc: devel@edk2.groups.io<mailto:devel@edk2.groups.io> >>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>> disable safe string constraint assertions >>> >>> Hi Mike, >>> >>> Yes, the primary use case is for UEFI Applications. We >>> do not want to disable ASSERT’s completely, as >>> assertions that make sense, i.e. the ones signalising >>> about interface misuse, are helpful for debugging. >>> >>> I have already explained in the BZ that basically all >>> safe string constraint assertions make no sense for >>> handling untrusted data. We find this use case very >>> logical, as these functions behave properly with >>> assertions disabled and cover all these error >>> conditions by the return statuses. In such situation is >>> not useful for these functions to assert, as we end up >>> inefficiently reimplementing the logic. I would have >>> liked the approach of discussing the interfaces >>> individually, but I struggle to find any that makes >>> sense from this point of view. >>> >>> AsciiStrToGuid will ASSERT when the length of the >>> passed string is odd. Functions that cannot, ahem, >>> parse, for us are pretty much useless. >>> AsciiStrCatS will ASSERT when the appended string does >>> not fit the buffer. For us this logic makes this >>> function pretty much equivalent to deprecated and thus >>> unavailable AsciiStrCat, except it is also slower. >>> >>> My original suggestion was to remove the assertions >>> entirely, but several people here said that they use >>> them to verify usage errors when handling trusted data. >>> This makes good sense to me, so we suggest to support >>> both cases by introducing a PCD in this patch. >>> >>> Best wishes, >>> Vitaly >>> >>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >>> <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> написал(а): >>>> >>>> >>>> Hi Vitaly, >>>> >>>> Is the use case for UEFI Applications? >>>> >>>> There is a different mechanism to disable all >>> ASSERT() >>>> statements within a UEFI Application. >>>> >>>> If a component is consuming data from an untrusted >>> source, >>>> then that component is required to verify the >>> untrusted >>>> data before passing it to a function that clearly >>> documents >>>> is input requirements. If this approach is followed, >>> then >>>> the BaseLib functions can be used "as is" as long as >>> the >>>> ASSERT() conditions are verified before calling. >>>> >>>> If there are some APIs that currently document their >>> ASSERT() >>>> behavior and we think that ASSERT() behavior is >>> incorrect and >>>> should be handled by an existing error return value, >>> then we >>>> should discuss each of those APIs individually. >>>> >>>> Mike >>>> >>>> >>>>> -----Original Message----- >>>>> From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On >>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>> Sent: Friday, January 3, 2020 9:13 AM >>>>> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> >>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >>> disable >>>>> safe string constraint assertions >>>>> >>>>> REF: >>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >>>>> >>>>> Requesting for merge in edk2-stable202002. >>>>> >>>>> Changes since V1: >>>>> - Enable assertions by default to preserve the >>> original >>>>> behaviour >>>>> - Fix bugzilla reference link >>>>> - Update documentation in BaseLib.h >>>>> >>>>> Vitaly Cheptsov (1): >>>>> MdePkg: Add PCD to disable safe string constraint >>>>> assertions >>>>> >>>>> MdePkg/MdePkg.dec | 6 ++ >>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >>>>> MdePkg/Include/Library/BaseLib.h | 74 >>>>> +++++++++++++------- >>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >>>>> MdePkg/MdePkg.uni | 6 ++ >>>>> 5 files changed, 71 insertions(+), 30 deletions(-) >>>>> >>>>> -- >>>>> 2.21.0 (Apple Git-122.2) >>>>> >>>>> >>>>> -=-=-=-=-=-= >>>>> Groups.io Links: You receive all messages sent to >>> this >>>>> group. >>>>> >>>>> View/Reply Online (#52837): >>>>> https://edk2.groups.io/g/devel/message/52837 >>>>> Mute This Topic: >>> https://groups.io/mt/69401948/1643496 >>>>> Group Owner: devel+owner@edk2.groups.io<mailto:devel+owner@edk2.groups.io> >>>>> Unsubscribe: https://edk2.groups.io/g/devel/unsub >>>>> [michael.d.kinney@intel.com] >>>>> -=-=-=-=-=-= >>>> >>> >>> >>> >> > [-- Attachment #2: Type: text/html, Size: 54368 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-14 17:00 ` Michael D Kinney @ 2020-02-14 17:37 ` Vitaly Cheptsov 2020-02-14 22:50 ` Michael D Kinney 0 siblings, 1 reply; 28+ messages in thread From: Vitaly Cheptsov @ 2020-02-14 17:37 UTC (permalink / raw) To: Kinney, Michael D Cc: devel@edk2.groups.io, Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek [-- Attachment #1.1: Type: text/plain, Size: 9454 bytes --] Michael, Generalising the approach makes good sense to me, but we need to make an obvious distinguishable difference between: - precondition and invariant assertions (i.e. assertions, where function will NOT work if they are violated) - constraint asserts (i.e. assertions, which allow us to spot unintentional behaviour when parsing untrusted data, but which do not break function behaviour). As we want to use this outside of SafeString, I suggest the following: - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED 0x40 bit for PcdDebugPropertyMask instead of PcdAssertOnSafeStringConstraints. - Introduce DebugAssertConstraintEnabled DebugLib function to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. - Introduce ASSERT_CONSTRAINT macro, that will assert only if DebugConstraintAssertEnabled returns true. - Change SafeString ASSERTS in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. - Use ASSERT_CONSTRAINT in other places where necessary. I believe this way lines best with EDK II design. If there are no objections, I can submit the patch in the beginning of next week. Best wishes, Vitaly > 14 февр. 2020 г., в 20:00, Kinney, Michael D <michael.d.kinney@intel.com> написал(а): > > Vitaly, > > I want to make sure a feature PCD can be used to disable ASSERT() behavior in more than just safe string functions in BaseLib. > > Can we consider changing the name and description of PcdAssertOnSafeStringConstraints to be more generic, so if we find other lib APIs, the name will make sense? > > Maybe something like: PcdEnableLibraryAssertChecks? Default is TRUE. Can set to FALSE in DSC file to disable ASSERT() checks. > > Thanks, > > Mike > > > > From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On Behalf Of Vitaly Cheptsov via Groups.Io > Sent: Friday, February 14, 2020 3:55 AM > To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>; Gao, Liming <liming.gao@intel.com <mailto:liming.gao@intel.com>>; Gao, Zhichao <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>; devel@edk2.groups.io <mailto:devel@edk2.groups.io> > Cc: Marvin Häuser <marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>> > Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions > > Replying as per Liming's request for this to be merged into edk2-stable202002. > > On Mon, Feb 10, 2020 at 14:12, vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> wrote: > Hello, > > It has been quite some time since we submitted the patch with so far no negative response. As I mentioned previously, my team will strongly benefit from its landing in EDK II mainline. Since it does not add any regressions and can be viewed as a feature implementation for the rest of EDK II users, I request this to be merged upstream in edk2-stable202002. > > Best wishes, > Vitaly > > > 27 янв. 2020 г., в 12:47, vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> написал(а): > > > > > > Hi Mike, > > > > Any progress with this? We would really benefit from this landing in the next stable release. > > > > Best, > > Vitaly > > > >> 8 янв. 2020 г., в 19:35, Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> написал(а): > >> > >> > >> Hi Vitaly, > >> > >> Thanks for the additional background. I would like > >> a couple extra day to review the PCD name and the places > >> the PCD might potentially be used. > >> > >> If we find other APIs where ASSERT() behavior is only > >> valuable during dev/debug to quickly identify misuse > >> with trusted data and the API provides predicable > >> return behavior when ASSERT() is disabled, then I would > >> like to have a pattern we can potentially apply to all > >> these APIs across all packages. > >> > >> Thanks, > >> > >> Mike > >> > >>> -----Original Message----- > >>> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On > >>> Behalf Of Vitaly Cheptsov via Groups.Io > >>> Sent: Monday, January 6, 2020 10:44 AM > >>> To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> > >>> Cc: devel@edk2.groups.io <mailto:devel@edk2.groups.io> > >>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to > >>> disable safe string constraint assertions > >>> > >>> Hi Mike, > >>> > >>> Yes, the primary use case is for UEFI Applications. We > >>> do not want to disable ASSERT’s completely, as > >>> assertions that make sense, i.e. the ones signalising > >>> about interface misuse, are helpful for debugging. > >>> > >>> I have already explained in the BZ that basically all > >>> safe string constraint assertions make no sense for > >>> handling untrusted data. We find this use case very > >>> logical, as these functions behave properly with > >>> assertions disabled and cover all these error > >>> conditions by the return statuses. In such situation is > >>> not useful for these functions to assert, as we end up > >>> inefficiently reimplementing the logic. I would have > >>> liked the approach of discussing the interfaces > >>> individually, but I struggle to find any that makes > >>> sense from this point of view. > >>> > >>> AsciiStrToGuid will ASSERT when the length of the > >>> passed string is odd. Functions that cannot, ahem, > >>> parse, for us are pretty much useless. > >>> AsciiStrCatS will ASSERT when the appended string does > >>> not fit the buffer. For us this logic makes this > >>> function pretty much equivalent to deprecated and thus > >>> unavailable AsciiStrCat, except it is also slower. > >>> > >>> My original suggestion was to remove the assertions > >>> entirely, but several people here said that they use > >>> them to verify usage errors when handling trusted data. > >>> This makes good sense to me, so we suggest to support > >>> both cases by introducing a PCD in this patch. > >>> > >>> Best wishes, > >>> Vitaly > >>> > >>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D > >>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> написал(а): > >>>> > >>>> > >>>> Hi Vitaly, > >>>> > >>>> Is the use case for UEFI Applications? > >>>> > >>>> There is a different mechanism to disable all > >>> ASSERT() > >>>> statements within a UEFI Application. > >>>> > >>>> If a component is consuming data from an untrusted > >>> source, > >>>> then that component is required to verify the > >>> untrusted > >>>> data before passing it to a function that clearly > >>> documents > >>>> is input requirements. If this approach is followed, > >>> then > >>>> the BaseLib functions can be used "as is" as long as > >>> the > >>>> ASSERT() conditions are verified before calling. > >>>> > >>>> If there are some APIs that currently document their > >>> ASSERT() > >>>> behavior and we think that ASSERT() behavior is > >>> incorrect and > >>>> should be handled by an existing error return value, > >>> then we > >>>> should discuss each of those APIs individually. > >>>> > >>>> Mike > >>>> > >>>> > >>>>> -----Original Message----- > >>>>> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On > >>>>> Behalf Of Vitaly Cheptsov via Groups.Io > >>>>> Sent: Friday, January 3, 2020 9:13 AM > >>>>> To: devel@edk2.groups.io <mailto:devel@edk2.groups.io> > >>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to > >>> disable > >>>>> safe string constraint assertions > >>>>> > >>>>> REF: > >>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 <https://bugzilla.tianocore.org/show_bug.cgi?id=2054> > >>>>> > >>>>> Requesting for merge in edk2-stable202002. > >>>>> > >>>>> Changes since V1: > >>>>> - Enable assertions by default to preserve the > >>> original > >>>>> behaviour > >>>>> - Fix bugzilla reference link > >>>>> - Update documentation in BaseLib.h > >>>>> > >>>>> Vitaly Cheptsov (1): > >>>>> MdePkg: Add PCD to disable safe string constraint > >>>>> assertions > >>>>> > >>>>> MdePkg/MdePkg.dec | 6 ++ > >>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- > >>>>> MdePkg/Include/Library/BaseLib.h | 74 > >>>>> +++++++++++++------- > >>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- > >>>>> MdePkg/MdePkg.uni | 6 ++ > >>>>> 5 files changed, 71 insertions(+), 30 deletions(-) > >>>>> > >>>>> -- > >>>>> 2.21.0 (Apple Git-122.2) > >>>>> > >>>>> > >>>>> -=-=-=-=-=-= > >>>>> Groups.io <http://groups.io/> Links: You receive all messages sent to > >>> this > >>>>> group. > >>>>> > >>>>> View/Reply Online (#52837): > >>>>> https://edk2.groups.io/g/devel/message/52837 <https://edk2.groups.io/g/devel/message/52837> > >>>>> Mute This Topic: > >>> https://groups.io/mt/69401948/1643496 <https://groups.io/mt/69401948/1643496> > >>>>> Group Owner: devel+owner@edk2.groups.io <mailto:devel+owner@edk2.groups.io> > >>>>> Unsubscribe: https://edk2.groups.io/g/devel/unsub <https://edk2.groups.io/g/devel/unsub> > >>>>> [michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>] > >>>>> -=-=-=-=-=-= > >>>> > >>> > >>> > >>> > >> > > > > > > [-- Attachment #1.2: Type: text/html, Size: 25785 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 489 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-14 17:37 ` Vitaly Cheptsov @ 2020-02-14 22:50 ` Michael D Kinney 2020-02-14 23:04 ` Vitaly Cheptsov 2020-02-15 0:02 ` Andrew Fish 0 siblings, 2 replies; 28+ messages in thread From: Michael D Kinney @ 2020-02-14 22:50 UTC (permalink / raw) To: vit9696, Kinney, Michael D Cc: devel@edk2.groups.io, Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek [-- Attachment #1: Type: text/plain, Size: 10156 bytes --] Hi Vitaly, I agree that this proposal makes a lot of sense. We recently added a new assert type called STATIC_ASSERT() for assert conditions that can be tested at build time. A new assert type for checks that can be removed and the API still guarantees that it fails gracefully with a proper return code is a good idea. Given we have STATIC_ASSERT(), how about naming the new macro CONSTRAINT_ASSERT()? We also want the default to be enabled. The current use of bit 0x40 in PcdDebugPropertyMask is always clear, so we want the asserts enabled when 0x40 is clear. We can change the name of the define bit to DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED so bit 0x40 needs to be set in PcdDebugPropertyMask to disable these types of asserts. This approach does require all the DebugLib implementations to be updated with the new DebugConstraintAssertDisabled() API. Mike From: vit9696 <vit9696@protonmail.com> Sent: Friday, February 14, 2020 9:38 AM To: Kinney, Michael D <michael.d.kinney@intel.com> Cc: devel@edk2.groups.io; Gao, Liming <liming.gao@intel.com>; Gao, Zhichao <zhichao.gao@intel.com>; Marvin Häuser <marvin.haeuser@outlook.com>; Laszlo Ersek <lersek@redhat.com> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions Michael, Generalising the approach makes good sense to me, but we need to make an obvious distinguishable difference between: - precondition and invariant assertions (i.e. assertions, where function will NOT work if they are violated) - constraint asserts (i.e. assertions, which allow us to spot unintentional behaviour when parsing untrusted data, but which do not break function behaviour). As we want to use this outside of SafeString, I suggest the following: - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED 0x40 bit for PcdDebugPropertyMask instead of PcdAssertOnSafeStringConstraints. - Introduce DebugAssertConstraintEnabled DebugLib function to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. - Introduce ASSERT_CONSTRAINT macro, that will assert only if DebugConstraintAssertEnabled returns true. - Change SafeString ASSERTS in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. - Use ASSERT_CONSTRAINT in other places where necessary. I believe this way lines best with EDK II design. If there are no objections, I can submit the patch in the beginning of next week. Best wishes, Vitaly 14 февр. 2020 г., в 20:00, Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> написал(а): Vitaly, I want to make sure a feature PCD can be used to disable ASSERT() behavior in more than just safe string functions in BaseLib. Can we consider changing the name and description of PcdAssertOnSafeStringConstraints to be more generic, so if we find other lib APIs, the name will make sense? Maybe something like: PcdEnableLibraryAssertChecks? Default is TRUE. Can set to FALSE in DSC file to disable ASSERT() checks. Thanks, Mike From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Vitaly Cheptsov via Groups.Io Sent: Friday, February 14, 2020 3:55 AM To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Gao, Zhichao <zhichao.gao@intel.com<mailto:zhichao.gao@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io> Cc: Marvin Häuser <marvin.haeuser@outlook.com<mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek <lersek@redhat.com<mailto:lersek@redhat.com>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions Replying as per Liming's request for this to be merged into edk2-stable202002. On Mon, Feb 10, 2020 at 14:12, vit9696 <vit9696@protonmail.com<mailto:vit9696@protonmail.com>> wrote: Hello, It has been quite some time since we submitted the patch with so far no negative response. As I mentioned previously, my team will strongly benefit from its landing in EDK II mainline. Since it does not add any regressions and can be viewed as a feature implementation for the rest of EDK II users, I request this to be merged upstream in edk2-stable202002. Best wishes, Vitaly > 27 янв. 2020 г., в 12:47, vit9696 <vit9696@protonmail.com<mailto:vit9696@protonmail.com>> написал(а): > > > Hi Mike, > > Any progress with this? We would really benefit from this landing in the next stable release. > > Best, > Vitaly > >> 8 янв. 2020 г., в 19:35, Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> написал(а): >> >> >> Hi Vitaly, >> >> Thanks for the additional background. I would like >> a couple extra day to review the PCD name and the places >> the PCD might potentially be used. >> >> If we find other APIs where ASSERT() behavior is only >> valuable during dev/debug to quickly identify misuse >> with trusted data and the API provides predicable >> return behavior when ASSERT() is disabled, then I would >> like to have a pattern we can potentially apply to all >> these APIs across all packages. >> >> Thanks, >> >> Mike >> >>> -----Original Message----- >>> From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On >>> Behalf Of Vitaly Cheptsov via Groups.Io >>> Sent: Monday, January 6, 2020 10:44 AM >>> To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> >>> Cc: devel@edk2.groups.io<mailto:devel@edk2.groups.io> >>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>> disable safe string constraint assertions >>> >>> Hi Mike, >>> >>> Yes, the primary use case is for UEFI Applications. We >>> do not want to disable ASSERT’s completely, as >>> assertions that make sense, i.e. the ones signalising >>> about interface misuse, are helpful for debugging. >>> >>> I have already explained in the BZ that basically all >>> safe string constraint assertions make no sense for >>> handling untrusted data. We find this use case very >>> logical, as these functions behave properly with >>> assertions disabled and cover all these error >>> conditions by the return statuses. In such situation is >>> not useful for these functions to assert, as we end up >>> inefficiently reimplementing the logic. I would have >>> liked the approach of discussing the interfaces >>> individually, but I struggle to find any that makes >>> sense from this point of view. >>> >>> AsciiStrToGuid will ASSERT when the length of the >>> passed string is odd. Functions that cannot, ahem, >>> parse, for us are pretty much useless. >>> AsciiStrCatS will ASSERT when the appended string does >>> not fit the buffer. For us this logic makes this >>> function pretty much equivalent to deprecated and thus >>> unavailable AsciiStrCat, except it is also slower. >>> >>> My original suggestion was to remove the assertions >>> entirely, but several people here said that they use >>> them to verify usage errors when handling trusted data. >>> This makes good sense to me, so we suggest to support >>> both cases by introducing a PCD in this patch. >>> >>> Best wishes, >>> Vitaly >>> >>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >>> <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> написал(а): >>>> >>>> >>>> Hi Vitaly, >>>> >>>> Is the use case for UEFI Applications? >>>> >>>> There is a different mechanism to disable all >>> ASSERT() >>>> statements within a UEFI Application. >>>> >>>> If a component is consuming data from an untrusted >>> source, >>>> then that component is required to verify the >>> untrusted >>>> data before passing it to a function that clearly >>> documents >>>> is input requirements. If this approach is followed, >>> then >>>> the BaseLib functions can be used "as is" as long as >>> the >>>> ASSERT() conditions are verified before calling. >>>> >>>> If there are some APIs that currently document their >>> ASSERT() >>>> behavior and we think that ASSERT() behavior is >>> incorrect and >>>> should be handled by an existing error return value, >>> then we >>>> should discuss each of those APIs individually. >>>> >>>> Mike >>>> >>>> >>>>> -----Original Message----- >>>>> From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On >>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>> Sent: Friday, January 3, 2020 9:13 AM >>>>> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> >>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >>> disable >>>>> safe string constraint assertions >>>>> >>>>> REF: >>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >>>>> >>>>> Requesting for merge in edk2-stable202002. >>>>> >>>>> Changes since V1: >>>>> - Enable assertions by default to preserve the >>> original >>>>> behaviour >>>>> - Fix bugzilla reference link >>>>> - Update documentation in BaseLib.h >>>>> >>>>> Vitaly Cheptsov (1): >>>>> MdePkg: Add PCD to disable safe string constraint >>>>> assertions >>>>> >>>>> MdePkg/MdePkg.dec | 6 ++ >>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >>>>> MdePkg/Include/Library/BaseLib.h | 74 >>>>> +++++++++++++------- >>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >>>>> MdePkg/MdePkg.uni | 6 ++ >>>>> 5 files changed, 71 insertions(+), 30 deletions(-) >>>>> >>>>> -- >>>>> 2.21.0 (Apple Git-122.2) >>>>> >>>>> >>>>> -=-=-=-=-=-= >>>>> Groups.io<http://groups.io/> Links: You receive all messages sent to >>> this >>>>> group. >>>>> >>>>> View/Reply Online (#52837): >>>>> https://edk2.groups.io/g/devel/message/52837 >>>>> Mute This Topic: >>> https://groups.io/mt/69401948/1643496 >>>>> Group Owner: devel+owner@edk2.groups.io<mailto:devel+owner@edk2.groups.io> >>>>> Unsubscribe: https://edk2.groups.io/g/devel/unsub >>>>> [michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>] >>>>> -=-=-=-=-=-= >>>> >>> >>> >>> >> > [-- Attachment #2: Type: text/html, Size: 67871 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-14 22:50 ` Michael D Kinney @ 2020-02-14 23:04 ` Vitaly Cheptsov 2020-02-15 0:02 ` Andrew Fish 1 sibling, 0 replies; 28+ messages in thread From: Vitaly Cheptsov @ 2020-02-14 23:04 UTC (permalink / raw) To: Kinney, Michael D Cc: devel@edk2.groups.io, Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek [-- Attachment #1: Type: text/plain, Size: 10276 bytes --] Michael, The suggested changes make sense to me. I will prepare the patch in the next days. I guess the only question left is whether disabling assertions also disables constraint assertions. I think this should be the case for backwards compatibility, despite being slightly unintuitive. Best, Vitaly On Sat, Feb 15, 2020 at 01:50, Kinney, Michael D <michael.d.kinney@intel.com> wrote: > Hi Vitaly, > > I agree that this proposal makes a lot of sense. We recently added a new assert type called STATIC_ASSERT() for assert conditions that can be tested at build time. > > A new assert type for checks that can be removed and the API still guarantees that it fails gracefully with a proper return code is a good idea. Given we have STATIC_ASSERT(), how about naming the new macro CONSTRAINT_ASSERT()? > > We also want the default to be enabled. The current use of bit 0x40 in PcdDebugPropertyMask is always clear, so we want the asserts enabled when 0x40 is clear. We can change the name of the define bit to DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED so bit 0x40 needs to be set in PcdDebugPropertyMask to disable these types of asserts. > > This approach does require all the DebugLib implementations to be updated with the new DebugConstraintAssertDisabled() API. > > Mike > > From: vit9696 <vit9696@protonmail.com> > Sent: Friday, February 14, 2020 9:38 AM > To: Kinney, Michael D <michael.d.kinney@intel.com> > Cc: devel@edk2.groups.io; Gao, Liming <liming.gao@intel.com>; Gao, Zhichao <zhichao.gao@intel.com>; Marvin Häuser <marvin.haeuser@outlook.com>; Laszlo Ersek <lersek@redhat.com> > Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions > > Michael, > > Generalising the approach makes good sense to me, but we need to make an obvious distinguishable difference between: > > - precondition and invariant assertions (i.e. assertions, where function will NOT work if they are violated) > > - constraint asserts (i.e. assertions, which allow us to spot unintentional behaviour when parsing untrusted data, but which do not break function behaviour). > > As we want to use this outside of SafeString, I suggest the following: > > - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED 0x40 bit for PcdDebugPropertyMask instead of PcdAssertOnSafeStringConstraints. > > - Introduce DebugAssertConstraintEnabled DebugLib function to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. > > - Introduce ASSERT_CONSTRAINT macro, that will assert only if DebugConstraintAssertEnabled returns true. > > - Change SafeString ASSERTS in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. > > - Use ASSERT_CONSTRAINT in other places where necessary. > > I believe this way lines best with EDK II design. If there are no objections, I can submit the patch in the beginning of next week. > > Best wishes, > > Vitaly > >> 14 февр. 2020 г., в 20:00, Kinney, Michael D <michael.d.kinney@intel.com> написал(а): >> >> Vitaly, >> >> I want to make sure a feature PCD can be used to disable ASSERT() behavior in more than just safe string functions in BaseLib. >> >> Can we consider changing the name and description of PcdAssertOnSafeStringConstraints to be more generic, so if we find other lib APIs, the name will make sense? >> >> Maybe something like: PcdEnableLibraryAssertChecks? Default is TRUE. Can set to FALSE in DSC file to disable ASSERT() checks. >> >> Thanks, >> >> Mike >> >> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Vitaly Cheptsov via Groups.Io >> Sent: Friday, February 14, 2020 3:55 AM >> To: Kinney, Michael D <michael.d.kinney@intel.com>; Gao, Liming <liming.gao@intel.com>; Gao, Zhichao <zhichao.gao@intel.com>; devel@edk2.groups.io >> Cc: Marvin Häuser <marvin.haeuser@outlook.com>; Laszlo Ersek <lersek@redhat.com> >> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions >> >> Replying as per Liming's request for this to be merged into edk2-stable202002. >> >> On Mon, Feb 10, 2020 at 14:12, vit9696 <vit9696@protonmail.com> wrote: >> >>> Hello, >>> >>> It has been quite some time since we submitted the patch with so far no negative response. As I mentioned previously, my team will strongly benefit from its landing in EDK II mainline. Since it does not add any regressions and can be viewed as a feature implementation for the rest of EDK II users, I request this to be merged upstream in edk2-stable202002. >>> >>> Best wishes, >>> Vitaly >>> >>>> 27 янв. 2020 г., в 12:47, vit9696 <vit9696@protonmail.com> написал(а): >>>> >>>> >>>> Hi Mike, >>>> >>>> Any progress with this? We would really benefit from this landing in the next stable release. >>>> >>>> Best, >>>> Vitaly >>>> >>>>> 8 янв. 2020 г., в 19:35, Kinney, Michael D <michael.d.kinney@intel.com> написал(а): >>>>> >>>>> >>>>> Hi Vitaly, >>>>> >>>>> Thanks for the additional background. I would like >>>>> a couple extra day to review the PCD name and the places >>>>> the PCD might potentially be used. >>>>> >>>>> If we find other APIs where ASSERT() behavior is only >>>>> valuable during dev/debug to quickly identify misuse >>>>> with trusted data and the API provides predicable >>>>> return behavior when ASSERT() is disabled, then I would >>>>> like to have a pattern we can potentially apply to all >>>>> these APIs across all packages. >>>>> >>>>> Thanks, >>>>> >>>>> Mike >>>>> >>>>>> -----Original Message----- >>>>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On >>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>> Sent: Monday, January 6, 2020 10:44 AM >>>>>> To: Kinney, Michael D <michael.d.kinney@intel.com> >>>>>> Cc: devel@edk2.groups.io >>>>>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>>>> disable safe string constraint assertions >>>>>> >>>>>> Hi Mike, >>>>>> >>>>>> Yes, the primary use case is for UEFI Applications. We >>>>>> do not want to disable ASSERT’s completely, as >>>>>> assertions that make sense, i.e. the ones signalising >>>>>> about interface misuse, are helpful for debugging. >>>>>> >>>>>> I have already explained in the BZ that basically all >>>>>> safe string constraint assertions make no sense for >>>>>> handling untrusted data. We find this use case very >>>>>> logical, as these functions behave properly with >>>>>> assertions disabled and cover all these error >>>>>> conditions by the return statuses. In such situation is >>>>>> not useful for these functions to assert, as we end up >>>>>> inefficiently reimplementing the logic. I would have >>>>>> liked the approach of discussing the interfaces >>>>>> individually, but I struggle to find any that makes >>>>>> sense from this point of view. >>>>>> >>>>>> AsciiStrToGuid will ASSERT when the length of the >>>>>> passed string is odd. Functions that cannot, ahem, >>>>>> parse, for us are pretty much useless. >>>>>> AsciiStrCatS will ASSERT when the appended string does >>>>>> not fit the buffer. For us this logic makes this >>>>>> function pretty much equivalent to deprecated and thus >>>>>> unavailable AsciiStrCat, except it is also slower. >>>>>> >>>>>> My original suggestion was to remove the assertions >>>>>> entirely, but several people here said that they use >>>>>> them to verify usage errors when handling trusted data. >>>>>> This makes good sense to me, so we suggest to support >>>>>> both cases by introducing a PCD in this patch. >>>>>> >>>>>> Best wishes, >>>>>> Vitaly >>>>>> >>>>>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >>>>>> <michael.d.kinney@intel.com> написал(а): >>>>>>> >>>>>>> >>>>>>> Hi Vitaly, >>>>>>> >>>>>>> Is the use case for UEFI Applications? >>>>>>> >>>>>>> There is a different mechanism to disable all >>>>>> ASSERT() >>>>>>> statements within a UEFI Application. >>>>>>> >>>>>>> If a component is consuming data from an untrusted >>>>>> source, >>>>>>> then that component is required to verify the >>>>>> untrusted >>>>>>> data before passing it to a function that clearly >>>>>> documents >>>>>>> is input requirements. If this approach is followed, >>>>>> then >>>>>>> the BaseLib functions can be used "as is" as long as >>>>>> the >>>>>>> ASSERT() conditions are verified before calling. >>>>>>> >>>>>>> If there are some APIs that currently document their >>>>>> ASSERT() >>>>>>> behavior and we think that ASSERT() behavior is >>>>>> incorrect and >>>>>>> should be handled by an existing error return value, >>>>>> then we >>>>>>> should discuss each of those APIs individually. >>>>>>> >>>>>>> Mike >>>>>>> >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On >>>>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>>>> Sent: Friday, January 3, 2020 9:13 AM >>>>>>>> To: devel@edk2.groups.io >>>>>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>>>> disable >>>>>>>> safe string constraint assertions >>>>>>>> >>>>>>>> REF: >>>>>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >>>>>>>> >>>>>>>> Requesting for merge in edk2-stable202002. >>>>>>>> >>>>>>>> Changes since V1: >>>>>>>> - Enable assertions by default to preserve the >>>>>> original >>>>>>>> behaviour >>>>>>>> - Fix bugzilla reference link >>>>>>>> - Update documentation in BaseLib.h >>>>>>>> >>>>>>>> Vitaly Cheptsov (1): >>>>>>>> MdePkg: Add PCD to disable safe string constraint >>>>>>>> assertions >>>>>>>> >>>>>>>> MdePkg/MdePkg.dec | 6 ++ >>>>>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >>>>>>>> MdePkg/Include/Library/BaseLib.h | 74 >>>>>>>> +++++++++++++------- >>>>>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >>>>>>>> MdePkg/MdePkg.uni | 6 ++ >>>>>>>> 5 files changed, 71 insertions(+), 30 deletions(-) >>>>>>>> >>>>>>>> -- >>>>>>>> 2.21.0 (Apple Git-122.2) >>>>>>>> >>>>>>>> >>>>>>>> -=-=-=-=-=-= >>>>>>>> [Groups.io](http://groups.io/) Links: You receive all messages sent to >>>>>> this >>>>>>>> group. >>>>>>>> >>>>>>>> View/Reply Online (#52837): >>>>>>>> https://edk2.groups.io/g/devel/message/52837 >>>>>>>> Mute This Topic: >>>>>> https://groups.io/mt/69401948/1643496 >>>>>>>> Group Owner: devel+owner@edk2.groups.io >>>>>>>> Unsubscribe: https://edk2.groups.io/g/devel/unsub >>>>>>>> [michael.d.kinney@intel.com] >>>>>>>> -=-=-=-=-=-= >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >> >> [-- Attachment #2: Type: text/html, Size: 29986 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-14 22:50 ` Michael D Kinney 2020-02-14 23:04 ` Vitaly Cheptsov @ 2020-02-15 0:02 ` Andrew Fish 2020-02-15 3:31 ` Vitaly Cheptsov 1 sibling, 1 reply; 28+ messages in thread From: Andrew Fish @ 2020-02-15 0:02 UTC (permalink / raw) To: devel, Mike Kinney Cc: vit9696, Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek [-- Attachment #1: Type: text/plain, Size: 13971 bytes --] > On Feb 14, 2020, at 2:50 PM, Michael D Kinney <michael.d.kinney@intel.com> wrote: > > Hi Vitaly, > > I agree that this proposal makes a lot of sense. We recently added a new assert type called STATIC_ASSERT() for assert conditions that can be tested at build time. > > A new assert type for checks that can be removed and the API still guarantees that it fails gracefully with a proper return code is a good idea. Given we have STATIC_ASSERT(), how about naming the new macro CONSTRAINT_ASSERT()? > > We also want the default to be enabled. The current use of bit 0x40 in PcdDebugPropertyMask is always clear, so we want the asserts enabled when 0x40 is clear. We can change the name of the define bit to DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED so bit 0x40 needs to be set in PcdDebugPropertyMask to disable these types of asserts. > > This approach does require all the DebugLib implementations to be updated with the new DebugConstraintAssertDisabled() API. > Mike, If you wanted to be backward compatible you could just use DebugAssertEnabled () but in place of _ASSERT() use _CONSTRAINT_ASSERT #define _ASSERT(Expression) DebugAssert (__FILE__, __LINE__, #Expression) #define _CONSTRAINT_ASSERT(Expression) DebugPrint (DEBUG_ERROR, "ASSERT %a(%d): %a\n",, __FILE__, __LINE__, #Expression) Not as elegant as the non backward compatible change, but I thought I'd throw it out there. There are some ancient Apple C ASSERT macros [AssertMacros.h] that also have the concept of require. Require includes an exception label (goto label). It is like a CONSTRAINT_ASSERT() but with the label. On release builds the DEBUG prints are skipped. So we could do something like: EFI_STATUS Status = EFI_INVALID_PARAMETER; REQUIRE(Arg1 != NULL, ErrorExit); REQUIRE(Arg2 != NULL, ErrorExit); REQUIRE(Arg3 != NULL, ErrorExit); ErrorExit: return Status; There is another form that allows an ACTION (a statement to execute. So you could have: EFI_STATUS Status; REQUIRE_ACTION(Arg1 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); REQUIRE_ACTION(Arg2 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); REQUIRE_ACTION(Arg3 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); ErrorExit: return Status; If you use CONSTRAINT_ASSERT(); if (Arg1 == NULL || Arg2 == NULL || Arg3 == NULL) { CONSTRAINT_ASSERT (Arg1 != NULL); CONSTRAINT_ASSERT (Arg2 != NULL); CONSTRAINT_ASSERT (Arg3 != NULL); return EFI_INVALID_PARAMETER; } I'd note error processing args on entry is the simplest case. In a more complex case when cleanup is required the goto label is more useful. I guess we could argue for less typing and more symmetry and say use ASSERT, CONSTRAINT, and REQUIRE. I guess you could add an ASSERT_ACTION too. The AssertMacros.h versions also support _quiet (skip the print) and _string (add a string to the print) so you end up with: REQUIRE REQUIRE_STRING REQUIRE_QUIET REQUIRE_ACTION REQUIRE_ACTION_STRING REQUIRE_ACTION_QUIET We could also end up with CONSTRAINT CONSTRAINT_STRING CONSTRAINT_QUIET I think the main idea behind _QUIET is you can silence things that are too noisy, and you can easily make noise things show up by removing the _QUIET to debug. I'd thought I throw out the other forms for folks to think about. I'm probably biased as I used to looking at code and seeing things like require_action_string(Arg1 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER, "1st Arg1 check"); Thanks, Andrew Fish PS The old debug macros had 2 versions of CONSTRAINT check and verify. The check version was compiled out on a release build, the verify version always does the check and just skips the DEBUG print. > Mike > > > > From: vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> > Sent: Friday, February 14, 2020 9:38 AM > To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> > Cc: devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Gao, Liming <liming.gao@intel.com <mailto:liming.gao@intel.com>>; Gao, Zhichao <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>; Marvin Häuser <marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>> > Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions > > Michael, > > Generalising the approach makes good sense to me, but we need to make an obvious distinguishable difference between: > - precondition and invariant assertions (i.e. assertions, where function will NOT work if they are violated) > - constraint asserts (i.e. assertions, which allow us to spot unintentional behaviour when parsing untrusted data, but which do not break function behaviour). > > As we want to use this outside of SafeString, I suggest the following: > - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED 0x40 bit for PcdDebugPropertyMask instead of PcdAssertOnSafeStringConstraints. > - Introduce DebugAssertConstraintEnabled DebugLib function to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. > - Introduce ASSERT_CONSTRAINT macro, that will assert only if DebugConstraintAssertEnabled returns true. > - Change SafeString ASSERTS in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. > - Use ASSERT_CONSTRAINT in other places where necessary. > > > I believe this way lines best with EDK II design. If there are no objections, I can submit the patch in the beginning of next week. > > > Best wishes, > Vitaly > > > 14 февр. 2020 г., в 20:00, Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> написал(а): > > Vitaly, > > I want to make sure a feature PCD can be used to disable ASSERT() behavior in more than just safe string functions in BaseLib. > > Can we consider changing the name and description of PcdAssertOnSafeStringConstraints to be more generic, so if we find other lib APIs, the name will make sense? > > Maybe something like: PcdEnableLibraryAssertChecks? Default is TRUE. Can set to FALSE in DSC file to disable ASSERT() checks. > > Thanks, > > Mike > > > > From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On Behalf Of Vitaly Cheptsov via Groups.Io > Sent: Friday, February 14, 2020 3:55 AM > To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>; Gao, Liming <liming.gao@intel.com <mailto:liming.gao@intel.com>>; Gao, Zhichao <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>; devel@edk2.groups.io <mailto:devel@edk2.groups.io> > Cc: Marvin Häuser <marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>> > Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions > > Replying as per Liming's request for this to be merged into edk2-stable202002. > > On Mon, Feb 10, 2020 at 14:12, vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> wrote: > Hello, > > It has been quite some time since we submitted the patch with so far no negative response. As I mentioned previously, my team will strongly benefit from its landing in EDK II mainline. Since it does not add any regressions and can be viewed as a feature implementation for the rest of EDK II users, I request this to be merged upstream in edk2-stable202002. > > Best wishes, > Vitaly > > > 27 янв. 2020 г., в 12:47, vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> написал(а): > > > > > > Hi Mike, > > > > Any progress with this? We would really benefit from this landing in the next stable release. > > > > Best, > > Vitaly > > > >> 8 янв. 2020 г., в 19:35, Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> написал(а): > >> > >> > >> Hi Vitaly, > >> > >> Thanks for the additional background. I would like > >> a couple extra day to review the PCD name and the places > >> the PCD might potentially be used. > >> > >> If we find other APIs where ASSERT() behavior is only > >> valuable during dev/debug to quickly identify misuse > >> with trusted data and the API provides predicable > >> return behavior when ASSERT() is disabled, then I would > >> like to have a pattern we can potentially apply to all > >> these APIs across all packages. > >> > >> Thanks, > >> > >> Mike > >> > >>> -----Original Message----- > >>> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On > >>> Behalf Of Vitaly Cheptsov via Groups.Io > >>> Sent: Monday, January 6, 2020 10:44 AM > >>> To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> > >>> Cc: devel@edk2.groups.io <mailto:devel@edk2.groups.io> > >>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to > >>> disable safe string constraint assertions > >>> > >>> Hi Mike, > >>> > >>> Yes, the primary use case is for UEFI Applications. We > >>> do not want to disable ASSERT’s completely, as > >>> assertions that make sense, i.e. the ones signalising > >>> about interface misuse, are helpful for debugging. > >>> > >>> I have already explained in the BZ that basically all > >>> safe string constraint assertions make no sense for > >>> handling untrusted data. We find this use case very > >>> logical, as these functions behave properly with > >>> assertions disabled and cover all these error > >>> conditions by the return statuses. In such situation is > >>> not useful for these functions to assert, as we end up > >>> inefficiently reimplementing the logic. I would have > >>> liked the approach of discussing the interfaces > >>> individually, but I struggle to find any that makes > >>> sense from this point of view. > >>> > >>> AsciiStrToGuid will ASSERT when the length of the > >>> passed string is odd. Functions that cannot, ahem, > >>> parse, for us are pretty much useless. > >>> AsciiStrCatS will ASSERT when the appended string does > >>> not fit the buffer. For us this logic makes this > >>> function pretty much equivalent to deprecated and thus > >>> unavailable AsciiStrCat, except it is also slower. > >>> > >>> My original suggestion was to remove the assertions > >>> entirely, but several people here said that they use > >>> them to verify usage errors when handling trusted data. > >>> This makes good sense to me, so we suggest to support > >>> both cases by introducing a PCD in this patch. > >>> > >>> Best wishes, > >>> Vitaly > >>> > >>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D > >>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> написал(а): > >>>> > >>>> > >>>> Hi Vitaly, > >>>> > >>>> Is the use case for UEFI Applications? > >>>> > >>>> There is a different mechanism to disable all > >>> ASSERT() > >>>> statements within a UEFI Application. > >>>> > >>>> If a component is consuming data from an untrusted > >>> source, > >>>> then that component is required to verify the > >>> untrusted > >>>> data before passing it to a function that clearly > >>> documents > >>>> is input requirements. If this approach is followed, > >>> then > >>>> the BaseLib functions can be used "as is" as long as > >>> the > >>>> ASSERT() conditions are verified before calling. > >>>> > >>>> If there are some APIs that currently document their > >>> ASSERT() > >>>> behavior and we think that ASSERT() behavior is > >>> incorrect and > >>>> should be handled by an existing error return value, > >>> then we > >>>> should discuss each of those APIs individually. > >>>> > >>>> Mike > >>>> > >>>> > >>>>> -----Original Message----- > >>>>> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On > >>>>> Behalf Of Vitaly Cheptsov via Groups.Io > >>>>> Sent: Friday, January 3, 2020 9:13 AM > >>>>> To: devel@edk2.groups.io <mailto:devel@edk2.groups.io> > >>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to > >>> disable > >>>>> safe string constraint assertions > >>>>> > >>>>> REF: > >>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 <https://bugzilla.tianocore.org/show_bug.cgi?id=2054> > >>>>> > >>>>> Requesting for merge in edk2-stable202002. > >>>>> > >>>>> Changes since V1: > >>>>> - Enable assertions by default to preserve the > >>> original > >>>>> behaviour > >>>>> - Fix bugzilla reference link > >>>>> - Update documentation in BaseLib.h > >>>>> > >>>>> Vitaly Cheptsov (1): > >>>>> MdePkg: Add PCD to disable safe string constraint > >>>>> assertions > >>>>> > >>>>> MdePkg/MdePkg.dec | 6 ++ > >>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- > >>>>> MdePkg/Include/Library/BaseLib.h | 74 > >>>>> +++++++++++++------- > >>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- > >>>>> MdePkg/MdePkg.uni | 6 ++ > >>>>> 5 files changed, 71 insertions(+), 30 deletions(-) > >>>>> > >>>>> -- > >>>>> 2.21.0 (Apple Git-122.2) > >>>>> > >>>>> > >>>>> -=-=-=-=-=-= > >>>>> Groups.io <http://groups.io/> Links: You receive all messages sent to > >>> this > >>>>> group. > >>>>> > >>>>> View/Reply Online (#52837): > >>>>> https://edk2.groups.io/g/devel/message/52837 <https://edk2.groups.io/g/devel/message/52837> > >>>>> Mute This Topic: > >>> https://groups.io/mt/69401948/1643496 <https://groups.io/mt/69401948/1643496> > >>>>> Group Owner: devel+owner@edk2.groups.io <mailto:devel+owner@edk2.groups.io> > >>>>> Unsubscribe: https://edk2.groups.io/g/devel/unsub <https://edk2.groups.io/g/devel/unsub> > >>>>> [michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>] > >>>>> -=-=-=-=-=-= > >>>> > >>> > >>> > >>> > >> > > > > > > > [-- Attachment #2: Type: text/html, Size: 41400 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-15 0:02 ` Andrew Fish @ 2020-02-15 3:31 ` Vitaly Cheptsov 2020-02-15 6:26 ` Andrew Fish 0 siblings, 1 reply; 28+ messages in thread From: Vitaly Cheptsov @ 2020-02-15 3:31 UTC (permalink / raw) To: Andrew Fish, devel, Mike Kinney Cc: Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek [-- Attachment #1: Type: text/plain, Size: 15585 bytes --] Hi Andrew, While your suggestions look interesting, I am afraid they are not particularly what we want to cover with this discussion at the moment. Making assertions go through DEBUG printing functions/macros is what we have to strongly disagree about. Assertions and debug prints are separate things configurable by separate PCDs. We should not mix them. Introducing constraint assertions is a logical step forward in understanding that different software works in different environments. - There are normal, or, as I call them, invariant assertions (e.g. preconditions), for places where the function cannot work unless the assertion is satisfied. This is where we ASSERT. - There are constraint assertions, which signalise that bad data came through the function, even though the function was called from a trusted source. This is where we call CONSTRAINT_ASSERT. The thing we need is to have the latter separable and configurable, because not every piece of software works in a trusted environment. Other than that, constraint assertions, when enabled, are not anyhow different from normal assertions in the sense of action taken. Assertions have configurable breakpoints and deadloops, and DEBUG prints go through a different route in DebugLib that may cause entirely different effects. For example, we halt execution upon printing to DEBUG_ERROR in our DebugLib even in release builds. To make it clear, currently I plan to add the following interface: #define CONSTRAINT_ASSERT(Expression) \ do { \ if (DebugConstraintAssertEnabled ()) { \ if (!(Expression)) { \ _ASSERT (Expression); \ ANALYZER_UNREACHABLE (); \ } \ } \ } while (FALSE) with DebugConstraintAssertEnabled implemented as (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED | DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED) == DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED) Your suggestion with require macros looks interesting indeed, but I believe it is in fact parallel to this discussion. The change we discuss introduces a new assertion primitive — constraint assertions, while REQUIRE macros are mostly about advanced syntax sugar and higher level assertion primitives on top of existing ones. Perhaps we can have this and make a good use of it, especially given that it brought some practical benefit in Apple, but I would rather discuss this later once constraint assertions are merged into EDK II tree. Best wishes, Vitaly On Sat, Feb 15, 2020 at 03:02, Andrew Fish <afish@apple.com> wrote: >> On Feb 14, 2020, at 2:50 PM, Michael D Kinney <michael.d.kinney@intel.com> wrote: >> >> Hi Vitaly, >> >> I agree that this proposal makes a lot of sense. We recently added a new assert type called STATIC_ASSERT() for assert conditions that can be tested at build time. >> >> A new assert type for checks that can be removed and the API still guarantees that it fails gracefully with a proper return code is a good idea. Given we have STATIC_ASSERT(), how about naming the new macro CONSTRAINT_ASSERT()? >> >> We also want the default to be enabled. The current use of bit 0x40 in PcdDebugPropertyMask is always clear, so we want the asserts enabled when 0x40 is clear. We can change the name of the define bit to DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED so bit 0x40 needs to be set in PcdDebugPropertyMask to disable these types of asserts. >> >> This approach does require all the DebugLib implementations to be updated with the new DebugConstraintAssertDisabled() API. > > Mike, > > If you wanted to be backward compatible you could just use DebugAssertEnabled () but in place of _ASSERT() use _CONSTRAINT_ASSERT > > #define _ASSERT(Expression) DebugAssert (__FILE__, __LINE__, #Expression) > > #define _CONSTRAINT_ASSERT(Expression) DebugPrint (DEBUG_ERROR, "ASSERT %a(%d): %a\n",, __FILE__, __LINE__, #Expression) > > Not as elegant as the non backward compatible change, but I thought I'd throw it out there. > > There are some ancient Apple C ASSERT macros [AssertMacros.h] that also have the concept of require. Require includes an exception label (goto label). It is like a CONSTRAINT_ASSERT() but with the label. On release builds the DEBUG prints are skipped. > > So we could do something like: > > EFI_STATUS Status = EFI_INVALID_PARAMETER; > > REQUIRE(Arg1 != NULL, ErrorExit); > REQUIRE(Arg2 != NULL, ErrorExit); > REQUIRE(Arg3 != NULL, ErrorExit); > > ErrorExit: > return Status; > > There is another form that allows an ACTION (a statement to execute. So you could have: > > EFI_STATUS Status; > > REQUIRE_ACTION(Arg1 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); > REQUIRE_ACTION(Arg2 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); > REQUIRE_ACTION(Arg3 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); > > ErrorExit: > return Status; > > If you use CONSTRAINT_ASSERT(); > > if (Arg1 == NULL || Arg2 == NULL || Arg3 == NULL) { > CONSTRAINT_ASSERT (Arg1 != NULL); > CONSTRAINT_ASSERT (Arg2 != NULL); > CONSTRAINT_ASSERT (Arg3 != NULL); > return EFI_INVALID_PARAMETER; > } > > I'd note error processing args on entry is the simplest case. In a more complex case when cleanup is required the goto label is more useful. > > I guess we could argue for less typing and more symmetry and say use ASSERT, CONSTRAINT, and REQUIRE. I guess you could add an ASSERT_ACTION too. > > The AssertMacros.h versions also support _quiet (skip the print) and _string (add a string to the print) so you end up with: > REQUIRE > REQUIRE_STRING > > REQUIRE_QUIET > > REQUIRE_ACTION > > REQUIRE_ACTION_STRING > > REQUIRE_ACTION_QUIET > > We could also end up with > CONSTRAINT > CONSTRAINT_STRING > > CONSTRAINT_QUIET > > I think the main idea behind _QUIET is you can silence things that are too noisy, and you can easily make noise things show up by removing the _QUIET to debug. > > I'd thought I throw out the other forms for folks to think about. I'm probably biased as I used to looking at code and seeing things like require_action_string(Arg1 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER, "1st Arg1 check"); > > Thanks, > > Andrew Fish > > PS The old debug macros had 2 versions of CONSTRAINT check and verify. The check version was compiled out on a release build, the verify version always does the check and just skips the DEBUG print. > >> Mike >> >> From: vit9696 <vit9696@protonmail.com> >> Sent: Friday, February 14, 2020 9:38 AM >> To: Kinney, Michael D <michael.d.kinney@intel.com> >> Cc: devel@edk2.groups.io; Gao, Liming <liming.gao@intel.com>; Gao, Zhichao <zhichao.gao@intel.com>; Marvin Häuser <marvin.haeuser@outlook.com>; Laszlo Ersek <lersek@redhat.com> >> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions >> >> Michael, >> >> Generalising the approach makes good sense to me, but we need to make an obvious distinguishable difference between: >> - precondition and invariant assertions (i.e. assertions, where function will NOT work if they are violated) >> - constraint asserts (i.e. assertions, which allow us to spot unintentional behaviour when parsing untrusted data, but which do not break function behaviour). >> >> As we want to use this outside of SafeString, I suggest the following: >> - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED 0x40 bit for PcdDebugPropertyMask instead of PcdAssertOnSafeStringConstraints. >> - Introduce DebugAssertConstraintEnabled DebugLib function to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. >> - Introduce ASSERT_CONSTRAINT macro, that will assert only if DebugConstraintAssertEnabled returns true. >> - Change SafeString ASSERTS in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. >> - Use ASSERT_CONSTRAINT in other places where necessary. >> >> I believe this way lines best with EDK II design. If there are no objections, I can submit the patch in the beginning of next week. >> >> Best wishes, >> Vitaly >> >>> 14 февр. 2020 г., в 20:00, Kinney, Michael D <michael.d.kinney@intel.com> написал(а): >>> >>> Vitaly, >>> >>> I want to make sure a feature PCD can be used to disable ASSERT() behavior in more than just safe string functions in BaseLib. >>> >>> Can we consider changing the name and description of PcdAssertOnSafeStringConstraints to be more generic, so if we find other lib APIs, the name will make sense? >>> >>> Maybe something like: PcdEnableLibraryAssertChecks? Default is TRUE. Can set to FALSE in DSC file to disable ASSERT() checks. >>> >>> Thanks, >>> >>> Mike >>> >>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Vitaly Cheptsov via Groups.Io >>> Sent: Friday, February 14, 2020 3:55 AM >>> To: Kinney, Michael D <michael.d.kinney@intel.com>; Gao, Liming <liming.gao@intel.com>; Gao, Zhichao <zhichao.gao@intel.com>; devel@edk2.groups.io >>> Cc: Marvin Häuser <marvin.haeuser@outlook.com>; Laszlo Ersek <lersek@redhat.com> >>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions >>> >>> Replying as per Liming's request for this to be merged into edk2-stable202002. >>> >>> On Mon, Feb 10, 2020 at 14:12, vit9696 <vit9696@protonmail.com> wrote: >>> >>>> Hello, >>>> >>>> It has been quite some time since we submitted the patch with so far no negative response. As I mentioned previously, my team will strongly benefit from its landing in EDK II mainline. Since it does not add any regressions and can be viewed as a feature implementation for the rest of EDK II users, I request this to be merged upstream in edk2-stable202002. >>>> >>>> Best wishes, >>>> Vitaly >>>> >>>>> 27 янв. 2020 г., в 12:47, vit9696 <vit9696@protonmail.com> написал(а): >>>>> >>>>> >>>>> Hi Mike, >>>>> >>>>> Any progress with this? We would really benefit from this landing in the next stable release. >>>>> >>>>> Best, >>>>> Vitaly >>>>> >>>>>> 8 янв. 2020 г., в 19:35, Kinney, Michael D <michael.d.kinney@intel.com> написал(а): >>>>>> >>>>>> >>>>>> Hi Vitaly, >>>>>> >>>>>> Thanks for the additional background. I would like >>>>>> a couple extra day to review the PCD name and the places >>>>>> the PCD might potentially be used. >>>>>> >>>>>> If we find other APIs where ASSERT() behavior is only >>>>>> valuable during dev/debug to quickly identify misuse >>>>>> with trusted data and the API provides predicable >>>>>> return behavior when ASSERT() is disabled, then I would >>>>>> like to have a pattern we can potentially apply to all >>>>>> these APIs across all packages. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Mike >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On >>>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>>> Sent: Monday, January 6, 2020 10:44 AM >>>>>>> To: Kinney, Michael D <michael.d.kinney@intel.com> >>>>>>> Cc: devel@edk2.groups.io >>>>>>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>>>>> disable safe string constraint assertions >>>>>>> >>>>>>> Hi Mike, >>>>>>> >>>>>>> Yes, the primary use case is for UEFI Applications. We >>>>>>> do not want to disable ASSERT’s completely, as >>>>>>> assertions that make sense, i.e. the ones signalising >>>>>>> about interface misuse, are helpful for debugging. >>>>>>> >>>>>>> I have already explained in the BZ that basically all >>>>>>> safe string constraint assertions make no sense for >>>>>>> handling untrusted data. We find this use case very >>>>>>> logical, as these functions behave properly with >>>>>>> assertions disabled and cover all these error >>>>>>> conditions by the return statuses. In such situation is >>>>>>> not useful for these functions to assert, as we end up >>>>>>> inefficiently reimplementing the logic. I would have >>>>>>> liked the approach of discussing the interfaces >>>>>>> individually, but I struggle to find any that makes >>>>>>> sense from this point of view. >>>>>>> >>>>>>> AsciiStrToGuid will ASSERT when the length of the >>>>>>> passed string is odd. Functions that cannot, ahem, >>>>>>> parse, for us are pretty much useless. >>>>>>> AsciiStrCatS will ASSERT when the appended string does >>>>>>> not fit the buffer. For us this logic makes this >>>>>>> function pretty much equivalent to deprecated and thus >>>>>>> unavailable AsciiStrCat, except it is also slower. >>>>>>> >>>>>>> My original suggestion was to remove the assertions >>>>>>> entirely, but several people here said that they use >>>>>>> them to verify usage errors when handling trusted data. >>>>>>> This makes good sense to me, so we suggest to support >>>>>>> both cases by introducing a PCD in this patch. >>>>>>> >>>>>>> Best wishes, >>>>>>> Vitaly >>>>>>> >>>>>>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >>>>>>> <michael.d.kinney@intel.com> написал(а): >>>>>>>> >>>>>>>> >>>>>>>> Hi Vitaly, >>>>>>>> >>>>>>>> Is the use case for UEFI Applications? >>>>>>>> >>>>>>>> There is a different mechanism to disable all >>>>>>> ASSERT() >>>>>>>> statements within a UEFI Application. >>>>>>>> >>>>>>>> If a component is consuming data from an untrusted >>>>>>> source, >>>>>>>> then that component is required to verify the >>>>>>> untrusted >>>>>>>> data before passing it to a function that clearly >>>>>>> documents >>>>>>>> is input requirements. If this approach is followed, >>>>>>> then >>>>>>>> the BaseLib functions can be used "as is" as long as >>>>>>> the >>>>>>>> ASSERT() conditions are verified before calling. >>>>>>>> >>>>>>>> If there are some APIs that currently document their >>>>>>> ASSERT() >>>>>>>> behavior and we think that ASSERT() behavior is >>>>>>> incorrect and >>>>>>>> should be handled by an existing error return value, >>>>>>> then we >>>>>>>> should discuss each of those APIs individually. >>>>>>>> >>>>>>>> Mike >>>>>>>> >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On >>>>>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>>>>> Sent: Friday, January 3, 2020 9:13 AM >>>>>>>>> To: devel@edk2.groups.io >>>>>>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>>>>> disable >>>>>>>>> safe string constraint assertions >>>>>>>>> >>>>>>>>> REF: >>>>>>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >>>>>>>>> >>>>>>>>> Requesting for merge in edk2-stable202002. >>>>>>>>> >>>>>>>>> Changes since V1: >>>>>>>>> - Enable assertions by default to preserve the >>>>>>> original >>>>>>>>> behaviour >>>>>>>>> - Fix bugzilla reference link >>>>>>>>> - Update documentation in BaseLib.h >>>>>>>>> >>>>>>>>> Vitaly Cheptsov (1): >>>>>>>>> MdePkg: Add PCD to disable safe string constraint >>>>>>>>> assertions >>>>>>>>> >>>>>>>>> MdePkg/MdePkg.dec | 6 ++ >>>>>>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >>>>>>>>> MdePkg/Include/Library/BaseLib.h | 74 >>>>>>>>> +++++++++++++------- >>>>>>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >>>>>>>>> MdePkg/MdePkg.uni | 6 ++ >>>>>>>>> 5 files changed, 71 insertions(+), 30 deletions(-) >>>>>>>>> >>>>>>>>> -- >>>>>>>>> 2.21.0 (Apple Git-122.2) >>>>>>>>> >>>>>>>>> >>>>>>>>> -=-=-=-=-=-= >>>>>>>>> [Groups.io](http://groups.io/) Links: You receive all messages sent to >>>>>>> this >>>>>>>>> group. >>>>>>>>> >>>>>>>>> View/Reply Online (#52837): >>>>>>>>> https://edk2.groups.io/g/devel/message/52837 >>>>>>>>> Mute This Topic: >>>>>>> https://groups.io/mt/69401948/1643496 >>>>>>>>> Group Owner: devel+owner@edk2.groups.io >>>>>>>>> Unsubscribe: https://edk2.groups.io/g/devel/unsub >>>>>>>>> [michael.d.kinney@intel.com] >>>>>>>>> -=-=-=-=-=-= >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >> >> [-- Attachment #2: Type: text/html, Size: 50189 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-15 3:31 ` Vitaly Cheptsov @ 2020-02-15 6:26 ` Andrew Fish 2020-02-15 11:53 ` Vitaly Cheptsov 2020-02-15 19:38 ` Michael D Kinney 0 siblings, 2 replies; 28+ messages in thread From: Andrew Fish @ 2020-02-15 6:26 UTC (permalink / raw) To: vit9696 Cc: devel, Mike Kinney, Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek [-- Attachment #1: Type: text/plain, Size: 18943 bytes --] Vitaly, Sorry after I sent the mail I realized it may come across as me asking you to do work and that was not my intent. I will point out though that a non backward compatible change to something as fundamental as the DebugLib is a very big deal. I've got a few different custom implementations that would break with this change as Mike proposed. Given breaking every one's debug lib is such a big deal maybe it is something that we should do as a long term plan vs. some incremental fix. So my intent was to start a conversation about what else we might want to change if we are going to break the world. The only think worse than breaking the world is breaking the world frequently. I'm also a little worried that we are taking things that are today locally scoped like SAFE_STRING_CONSTRAINT_CHECK and SAFE_PRINT_CONSTRAINT_CHECK and making them global constructs. I think the way others have dealt with things like this is to make them be DEBUG prints vs. ASSERTs. Also even something as simple as SAFE_STRING_CONSTRAINT_CHECK could be called from code that wants ASSERT and CONSTRAINT_ASSERT behavior. It is not clear to me that the low level code knows the right thing to do in a global sense even if there is a PCD. It almost seems like we should have wrappers for the Safe string functions that implement the behavior you want as a caller. I'm not sure about that, but it seems like it is worth talking about? Thanks, Andrew Fish > On Feb 14, 2020, at 7:31 PM, vit9696 <vit9696@protonmail.com> wrote: > > Hi Andrew, > > While your suggestions look interesting, I am afraid they are not particularly what we want to cover with this discussion at the moment. > > Making assertions go through DEBUG printing functions/macros is what we have to strongly disagree about. Assertions and debug prints are separate things configurable by separate PCDs. We should not mix them. Introducing constraint assertions is a logical step forward in understanding that different software works in different environments. > There are normal, or, as I call them, invariant assertions (e.g. preconditions), for places where the function cannot work unless the assertion is satisfied. This is where we ASSERT. > There are constraint assertions, which signalise that bad data came through the function, even though the function was called from a trusted source. This is where we call CONSTRAINT_ASSERT. > The thing we need is to have the latter separable and configurable, because not every piece of software works in a trusted environment. Other than that, constraint assertions, when enabled, are not anyhow different from normal assertions in the sense of action taken. Assertions have configurable breakpoints and deadloops, and DEBUG prints go through a different route in DebugLib that may cause entirely different effects. For example, we halt execution upon printing to DEBUG_ERROR in our DebugLib even in release builds. > > =To make it clear, currently I plan to add the following interface: > > #define CONSTRAINT_ASSERT(Expression) \ > do { \ > if (DebugConstraintAssertEnabled ()) { \ > if (!(Expression)) { \ > _ASSERT (Expression); \ > ANALYZER_UNREACHABLE (); \ > } \ > } \ > } while (FALSE) > > with DebugConstraintAssertEnabled implemented as > > (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED | DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED) == DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED) > > Your suggestion with require macros looks interesting indeed, but I believe it is in fact parallel to this discussion. The change we discuss introduces a new assertion primitive — constraint assertions, while REQUIRE macros are mostly about advanced syntax sugar and higher level assertion primitives on top of existing ones. Perhaps we can have this and make a good use of it, especially given that it brought some practical benefit in Apple, but I would rather discuss this later once constraint assertions are merged into EDK II tree. > > Best wishes, > Vitaly > > On Sat, Feb 15, 2020 at 03:02, Andrew Fish <afish@apple.com <mailto:afish@apple.com>> wrote: >> >> >> >>> On Feb 14, 2020, at 2:50 PM, Michael D Kinney <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> wrote: >>> >>> Hi Vitaly, >>> >>> I agree that this proposal makes a lot of sense. We recently added a new assert type called STATIC_ASSERT() for assert conditions that can be tested at build time. >>> >>> A new assert type for checks that can be removed and the API still guarantees that it fails gracefully with a proper return code is a good idea. Given we have STATIC_ASSERT(), how about naming the new macro CONSTRAINT_ASSERT()? >>> >>> We also want the default to be enabled. The current use of bit 0x40 in PcdDebugPropertyMask is always clear, so we want the asserts enabled when 0x40 is clear. We can change the name of the define bit to DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED so bit 0x40 needs to be set in PcdDebugPropertyMask to disable these types of asserts. >>> >>> This approach does require all the DebugLib implementations to be updated with the new DebugConstraintAssertDisabled() API. >>> >> >> Mike, >> >> If you wanted to be backward compatible you could just use DebugAssertEnabled () but in place of _ASSERT() use _CONSTRAINT_ASSERT >> >> #define _ASSERT(Expression) DebugAssert (__FILE__, __LINE__, #Expression) >> >> #define _CONSTRAINT_ASSERT(Expression) DebugPrint (DEBUG_ERROR, "ASSERT %a(%d): %a\n",, __FILE__, __LINE__, #Expression) >> >> Not as elegant as the non backward compatible change, but I thought I'd throw it out there. >> >> There are some ancient Apple C ASSERT macros [AssertMacros.h] that also have the concept of require. Require includes an exception label (goto label). It is like a CONSTRAINT_ASSERT() but with the label. On release builds the DEBUG prints are skipped. >> >> So we could do something like: >> >> EFI_STATUS Status = EFI_INVALID_PARAMETER; >> >> REQUIRE(Arg1 != NULL, ErrorExit); >> REQUIRE(Arg2 != NULL, ErrorExit); >> REQUIRE(Arg3 != NULL, ErrorExit); >> >> ErrorExit: >> return Status; >> >> There is another form that allows an ACTION (a statement to execute. So you could have: >> >> EFI_STATUS Status; >> >> REQUIRE_ACTION(Arg1 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); >> REQUIRE_ACTION(Arg2 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); >> REQUIRE_ACTION(Arg3 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); >> >> ErrorExit: >> return Status; >> >> If you use CONSTRAINT_ASSERT(); >> >> if (Arg1 == NULL || Arg2 == NULL || Arg3 == NULL) { >> CONSTRAINT_ASSERT (Arg1 != NULL); >> CONSTRAINT_ASSERT (Arg2 != NULL); >> CONSTRAINT_ASSERT (Arg3 != NULL); >> return EFI_INVALID_PARAMETER; >> } >> >> I'd note error processing args on entry is the simplest case. In a more complex case when cleanup is required the goto label is more useful. >> >> I guess we could argue for less typing and more symmetry and say use ASSERT, CONSTRAINT, and REQUIRE. I guess you could add an ASSERT_ACTION too. >> >> The AssertMacros.h versions also support _quiet (skip the print) and _string (add a string to the print) so you end up with: >> REQUIRE >> REQUIRE_STRING >> REQUIRE_QUIET >> REQUIRE_ACTION >> REQUIRE_ACTION_STRING >> REQUIRE_ACTION_QUIET >> >> We could also end up with >> CONSTRAINT >> CONSTRAINT_STRING >> CONSTRAINT_QUIET >> >> I think the main idea behind _QUIET is you can silence things that are too noisy, and you can easily make noise things show up by removing the _QUIET to debug. >> >> I'd thought I throw out the other forms for folks to think about. I'm probably biased as I used to looking at code and seeing things like require_action_string(Arg1 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER, "1st Arg1 check"); >> >> Thanks, >> >> Andrew Fish >> >> PS The old debug macros had 2 versions of CONSTRAINT check and verify. The check version was compiled out on a release build, the verify version always does the check and just skips the DEBUG print. >> >>> Mike >>> >>> >>> >>> From: vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> >>> Sent: Friday, February 14, 2020 9:38 AM >>> To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> >>> Cc: devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Gao, Liming <liming.gao@intel.com <mailto:liming.gao@intel.com>>; Gao, Zhichao <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>; Marvin Häuser <marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>> >>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions >>> >>> Michael, >>> >>> Generalising the approach makes good sense to me, but we need to make an obvious distinguishable difference between: >>> - precondition and invariant assertions (i.e. assertions, where function will NOT work if they are violated) >>> - constraint asserts (i.e. assertions, which allow us to spot unintentional behaviour when parsing untrusted data, but which do not break function behaviour). >>> >>> As we want to use this outside of SafeString, I suggest the following: >>> - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED 0x40 bit for PcdDebugPropertyMask instead of PcdAssertOnSafeStringConstraints. >>> - Introduce DebugAssertConstraintEnabled DebugLib function to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. >>> - Introduce ASSERT_CONSTRAINT macro, that will assert only if DebugConstraintAssertEnabled returns true. >>> - Change SafeString ASSERTS in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. >>> - Use ASSERT_CONSTRAINT in other places where necessary. >>> >>> >>> I believe this way lines best with EDK II design. If there are no objections, I can submit the patch in the beginning of next week. >>> >>> >>> Best wishes, >>> Vitaly >>> >>> >>> 14 февр. 2020 г., в 20:00, Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> написал(а): >>> >>> Vitaly, >>> >>> I want to make sure a feature PCD can be used to disable ASSERT() behavior in more than just safe string functions in BaseLib. >>> >>> Can we consider changing the name and description of PcdAssertOnSafeStringConstraints to be more generic, so if we find other lib APIs, the name will make sense? >>> >>> Maybe something like: PcdEnableLibraryAssertChecks? Default is TRUE. Can set to FALSE in DSC file to disable ASSERT() checks. >>> >>> Thanks, >>> >>> Mike >>> >>> >>> >>> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On Behalf Of Vitaly Cheptsov via Groups.Io >>> Sent: Friday, February 14, 2020 3:55 AM >>> To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>; Gao, Liming <liming.gao@intel.com <mailto:liming.gao@intel.com>>; Gao, Zhichao <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>; devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>> Cc: Marvin Häuser <marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>> >>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions >>> >>> Replying as per Liming's request for this to be merged into edk2-stable202002. >>> >>> On Mon, Feb 10, 2020 at 14:12, vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> wrote: >>> Hello, >>> >>> It has been quite some time since we submitted the patch with so far no negative response. As I mentioned previously, my team will strongly benefit from its landing in EDK II mainline. Since it does not add any regressions and can be viewed as a feature implementation for the rest of EDK II users, I request this to be merged upstream in edk2-stable202002. >>> >>> Best wishes, >>> Vitaly >>> >>> > 27 янв. 2020 г., в 12:47, vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> написал(а): >>> > >>> > >>> > Hi Mike, >>> > >>> > Any progress with this? We would really benefit from this landing in the next stable release. >>> > >>> > Best, >>> > Vitaly >>> > >>> >> 8 янв. 2020 г., в 19:35, Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> написал(а): >>> >> >>> >> >>> >> Hi Vitaly, >>> >> >>> >> Thanks for the additional background. I would like >>> >> a couple extra day to review the PCD name and the places >>> >> the PCD might potentially be used. >>> >> >>> >> If we find other APIs where ASSERT() behavior is only >>> >> valuable during dev/debug to quickly identify misuse >>> >> with trusted data and the API provides predicable >>> >> return behavior when ASSERT() is disabled, then I would >>> >> like to have a pattern we can potentially apply to all >>> >> these APIs across all packages. >>> >> >>> >> Thanks, >>> >> >>> >> Mike >>> >> >>> >>> -----Original Message----- >>> >>> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On >>> >>> Behalf Of Vitaly Cheptsov via Groups.Io >>> >>> Sent: Monday, January 6, 2020 10:44 AM >>> >>> To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> >>> >>> Cc: devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>> >>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>> >>> disable safe string constraint assertions >>> >>> >>> >>> Hi Mike, >>> >>> >>> >>> Yes, the primary use case is for UEFI Applications. We >>> >>> do not want to disable ASSERT’s completely, as >>> >>> assertions that make sense, i.e. the ones signalising >>> >>> about interface misuse, are helpful for debugging. >>> >>> >>> >>> I have already explained in the BZ that basically all >>> >>> safe string constraint assertions make no sense for >>> >>> handling untrusted data. We find this use case very >>> >>> logical, as these functions behave properly with >>> >>> assertions disabled and cover all these error >>> >>> conditions by the return statuses. In such situation is >>> >>> not useful for these functions to assert, as we end up >>> >>> inefficiently reimplementing the logic. I would have >>> >>> liked the approach of discussing the interfaces >>> >>> individually, but I struggle to find any that makes >>> >>> sense from this point of view. >>> >>> >>> >>> AsciiStrToGuid will ASSERT when the length of the >>> >>> passed string is odd. Functions that cannot, ahem, >>> >>> parse, for us are pretty much useless. >>> >>> AsciiStrCatS will ASSERT when the appended string does >>> >>> not fit the buffer. For us this logic makes this >>> >>> function pretty much equivalent to deprecated and thus >>> >>> unavailable AsciiStrCat, except it is also slower. >>> >>> >>> >>> My original suggestion was to remove the assertions >>> >>> entirely, but several people here said that they use >>> >>> them to verify usage errors when handling trusted data. >>> >>> This makes good sense to me, so we suggest to support >>> >>> both cases by introducing a PCD in this patch. >>> >>> >>> >>> Best wishes, >>> >>> Vitaly >>> >>> >>> >>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >>> >>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> написал(а): >>> >>>> >>> >>>> >>> >>>> Hi Vitaly, >>> >>>> >>> >>>> Is the use case for UEFI Applications? >>> >>>> >>> >>>> There is a different mechanism to disable all >>> >>> ASSERT() >>> >>>> statements within a UEFI Application. >>> >>>> >>> >>>> If a component is consuming data from an untrusted >>> >>> source, >>> >>>> then that component is required to verify the >>> >>> untrusted >>> >>>> data before passing it to a function that clearly >>> >>> documents >>> >>>> is input requirements. If this approach is followed, >>> >>> then >>> >>>> the BaseLib functions can be used "as is" as long as >>> >>> the >>> >>>> ASSERT() conditions are verified before calling. >>> >>>> >>> >>>> If there are some APIs that currently document their >>> >>> ASSERT() >>> >>>> behavior and we think that ASSERT() behavior is >>> >>> incorrect and >>> >>>> should be handled by an existing error return value, >>> >>> then we >>> >>>> should discuss each of those APIs individually. >>> >>>> >>> >>>> Mike >>> >>>> >>> >>>> >>> >>>>> -----Original Message----- >>> >>>>> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On >>> >>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>> >>>>> Sent: Friday, January 3, 2020 9:13 AM >>> >>>>> To: devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>> >>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >>> >>> disable >>> >>>>> safe string constraint assertions >>> >>>>> >>> >>>>> REF: >>> >>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 <https://bugzilla.tianocore.org/show_bug.cgi?id=2054> >>> >>>>> >>> >>>>> Requesting for merge in edk2-stable202002. >>> >>>>> >>> >>>>> Changes since V1: >>> >>>>> - Enable assertions by default to preserve the >>> >>> original >>> >>>>> behaviour >>> >>>>> - Fix bugzilla reference link >>> >>>>> - Update documentation in BaseLib.h >>> >>>>> >>> >>>>> Vitaly Cheptsov (1): >>> >>>>> MdePkg: Add PCD to disable safe string constraint >>> >>>>> assertions >>> >>>>> >>> >>>>> MdePkg/MdePkg.dec | 6 ++ >>> >>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >>> >>>>> MdePkg/Include/Library/BaseLib.h | 74 >>> >>>>> +++++++++++++------- >>> >>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >>> >>>>> MdePkg/MdePkg.uni | 6 ++ >>> >>>>> 5 files changed, 71 insertions(+), 30 deletions(-) >>> >>>>> >>> >>>>> -- >>> >>>>> 2.21.0 (Apple Git-122.2) >>> >>>>> >>> >>>>> >>> >>>>> -=-=-=-=-=-= >>> >>>>> Groups.io <http://groups.io/> Links: You receive all messages sent to >>> >>> this >>> >>>>> group. >>> >>>>> >>> >>>>> View/Reply Online (#52837): >>> >>>>> https://edk2.groups.io/g/devel/message/52837 <https://edk2.groups.io/g/devel/message/52837> >>> >>>>> Mute This Topic: >>> >>> https://groups.io/mt/69401948/1643496 <https://groups.io/mt/69401948/1643496> >>> >>>>> Group Owner: devel+owner@edk2.groups.io <mailto:devel+owner@edk2.groups.io> >>> >>>>> Unsubscribe: https://edk2.groups.io/g/devel/unsub <https://edk2.groups.io/g/devel/unsub> >>> >>>>> [michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>] >>> >>>>> -=-=-=-=-=-= >>> >>>> >>> >>> >>> >>> >>> >>> >>> >> >>> > >>> >>> >>> >>> >>> >> > > [-- Attachment #2: Type: text/html, Size: 51450 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-15 6:26 ` Andrew Fish @ 2020-02-15 11:53 ` Vitaly Cheptsov 2020-02-15 12:02 ` Vitaly Cheptsov 2020-02-15 19:38 ` Michael D Kinney 1 sibling, 1 reply; 28+ messages in thread From: Vitaly Cheptsov @ 2020-02-15 11:53 UTC (permalink / raw) To: Andrew Fish Cc: devel, Mike Kinney, Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-15 11:53 ` Vitaly Cheptsov @ 2020-02-15 12:02 ` Vitaly Cheptsov 0 siblings, 0 replies; 28+ messages in thread From: Vitaly Cheptsov @ 2020-02-15 12:02 UTC (permalink / raw) To: Andrew Fish Cc: devel, Mike Kinney, Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek [-- Attachment #1.1: Type: text/plain, Size: 5317 bytes --] It seems that edk2.groups.io <http://edk2.groups.io/> has hit the limits for message size due to quotation levels (?), so I send another copy of my previous message to have it visible from the web-interface. Andrew, It is ok, as we are all here for mutual benefit, no worries. But you are right that we indeed want a solution for BZ 2054[1] to land as early as possible. It is great that after more than half a year :) we have some productive discussion, so I am all open for it. * The compliant about a breaking change in DebugLib is honestly fair, and I wonder whether we could avoid it. The reason it potentially breaks is the introduction of DebugConstraintAssertEnabled function that will need to land in every DebugLib implementation. I think we can look differently at this problem. Currently functions like DebugAssertEnabled, DebugPrintEnabled, DebugClearMemoryEnabled, and so on are copy-pasted from one DebugLib to another introducing almost 100 lines of exactly the same code every DebugLib implementation should write. It can be argued that one can implement these functions differently, but neither the description in DebugLib.h permits this, nor I have seen any implementation doing it so far. I believe we should check Pcd values directly in DebugLib.h header, which has a lot of pros including the backwards compatibility resolution: — reduction of code copy-pasting. — making compilers without LTO produce smaller binaries. — making third-party DebugLib implementations more flexible and easier to maintain. — resolving the problem of third-party DebugLib implementations not working with CONSTRAINT_ASSERT. To make it less intrusive we can provide an iterative approach: 1. Introduce Pcd checking macros like DEBUG_PRINT_ENABLED, DEBUG_ASSERT_ENABLED: #define DEBUG_PRINT_ENABLED() (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & DEBUG_PROPERTY_DEBUG_PRINT_ENABLED) != 0) 2. Switch DEBUG, ASSERT, and other related macros to use these new macros by replacing calls like DebugPrintEnabled() with macros like DEBUG_PRINT_ENABLED(). 3. Introduce DEBUG_CONSTRAINT_ASSERT_ENABLED and CONSTRAINT_ASSERT as discussed previously except DebugConstraintAssertEnabled is now a macro. 4. Remove DebugPrintEnabled and other functions from EDK II DebugLib implementations and header. Step 4 can be done through deprecation phase. However, I think these functions are not used anywhere but inside DebugLib implementations anyway, and the way they are structured should let still legacy third-party DebugLib implementations still having these functions to compile fine even without the immediate refactoring. * The point of AssertMacros and changing the world is valid, but to be frank with you I have an opinion that we should not incorporate this experience in EDK II. I do not know if these macros are used in Mac EFI code, as my only experience with them was during XNU kernelspace programming, but I do not have a memory of them bringing enough benefit. Basically, macro constructions that cannot be easily expressed with normal C functions, especially ones changing control flow with something like a goto, are unintuitive and overcomplicated. They make the code harder to read, especially to those not familiar with the rest of the codebase, for little to no reason. While there certainly may be some positive sides in these macros, like improved structural separation by introducing different categories of checks, I am afraid we do not need them as is. Even if we do implement something close in the future, I believe they should rather exist in a separate library and be an opt-in for newly contributed code. * The point of caller/callee control on the constraint violation is slightly tricky. We have already spent some time trying to nail it down, and I believe there is a good level of logic that can be split in two parts: 1. Assertion type choose. Which to write ASSERT or CONSTRAINT_ASSERT? You can rely on a simple thought experiment here. Does failing this assertion make the function work in the conditions it was not meant to be? I.e. will the function break or crash or will some other spec/doc-compliant implementations of this potentially do. When the answer is affirmative, it is an ASSERT, otherwise it is a CONSTRAINT_ASSERT. Examples in the BZ mentioning SafeString give a good grasp of the use cases. 2. Assertion handling choice. When do we want ASSERT or CONSTRAINT_ASSERT to be enabled? This is more likely what your question was about. First is that CONSTRAINT_ASSERTs only make sense to enable in DEBUG builds where ASSERTs are enabled. So the question comes to whether DEBUG builds should behave the same way as RELEASE builds functionality-wise or not. Let’s take an another simple thought experiment: should we halt in a build when external untrusted data theoretically exists and is not valid? If the answer is yes, then CONSTRAINT_ASSERT should be enabled in a DEBUG build, otherwise no. Basically you enable CONSTRAINT_ASSERTs when you e.g. debug your code trying to find a problem (let’s say a typo) and disable CONSTRAINT_ASSERTs when you give a DEBUG build to the end user for a more or less long-term usage. Sorry for the long read, but I hope I answered all the questions in consideration. Best wishes, Vitaly [-- Attachment #1.2: Type: text/html, Size: 9197 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 489 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-15 6:26 ` Andrew Fish 2020-02-15 11:53 ` Vitaly Cheptsov @ 2020-02-15 19:38 ` Michael D Kinney 2020-02-16 21:25 ` Andrew Fish 1 sibling, 1 reply; 28+ messages in thread From: Michael D Kinney @ 2020-02-15 19:38 UTC (permalink / raw) To: afish@apple.com, vit9696, Kinney, Michael D Cc: devel@edk2.groups.io, Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek [-- Attachment #1: Type: text/plain, Size: 18950 bytes --] Andrew, I do not think of this as a globally scoped PCD. It can be set in global scope in DSC. But is can also be set to values based on module type or for specific modules. In the case of the safe string functions, I think there is a desire to disable the constraint asserts when building a UEFI App or UEFI Driver and implement those modules to handle the error return values. Enabling the constraint asserts for PEI Core, DXE Core, SMM/MM Core, PEIM, DXE, SMM/MM modules makes sense to find incorrect input to these functions from modules that can guarantee the inputs would never return an error and catch these as part of dev/debug/validation builds. I would not expect disabling on a module by module basis to be common. I think the rule for API implementations is to only use CONSTRAINT_ASSERT() for conditions that are also checked and return an error or fail with predicable behavior that allows the system to continue to function. ASSERT() is for conditions that the system can not continue. Best regards, Mike From: afish@apple.com <afish@apple.com> Sent: Friday, February 14, 2020 10:27 PM To: vit9696 <vit9696@protonmail.com> Cc: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>; Gao, Liming <liming.gao@intel.com>; Gao, Zhichao <zhichao.gao@intel.com>; Marvin Häuser <marvin.haeuser@outlook.com>; Laszlo Ersek <lersek@redhat.com> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions Vitaly, Sorry after I sent the mail I realized it may come across as me asking you to do work and that was not my intent. I will point out though that a non backward compatible change to something as fundamental as the DebugLib is a very big deal. I've got a few different custom implementations that would break with this change as Mike proposed. Given breaking every one's debug lib is such a big deal maybe it is something that we should do as a long term plan vs. some incremental fix. So my intent was to start a conversation about what else we might want to change if we are going to break the world. The only think worse than breaking the world is breaking the world frequently. I'm also a little worried that we are taking things that are today locally scoped like SAFE_STRING_CONSTRAINT_CHECK and SAFE_PRINT_CONSTRAINT_CHECK and making them global constructs. I think the way others have dealt with things like this is to make them be DEBUG prints vs. ASSERTs. Also even something as simple as SAFE_STRING_CONSTRAINT_CHECK could be called from code that wants ASSERT and CONSTRAINT_ASSERT behavior. It is not clear to me that the low level code knows the right thing to do in a global sense even if there is a PCD. It almost seems like we should have wrappers for the Safe string functions that implement the behavior you want as a caller. I'm not sure about that, but it seems like it is worth talking about? Thanks, Andrew Fish On Feb 14, 2020, at 7:31 PM, vit9696 <vit9696@protonmail.com<mailto:vit9696@protonmail.com>> wrote: Hi Andrew, While your suggestions look interesting, I am afraid they are not particularly what we want to cover with this discussion at the moment. Making assertions go through DEBUG printing functions/macros is what we have to strongly disagree about. Assertions and debug prints are separate things configurable by separate PCDs. We should not mix them. Introducing constraint assertions is a logical step forward in understanding that different software works in different environments. * There are normal, or, as I call them, invariant assertions (e.g. preconditions), for places where the function cannot work unless the assertion is satisfied. This is where we ASSERT. * There are constraint assertions, which signalise that bad data came through the function, even though the function was called from a trusted source. This is where we call CONSTRAINT_ASSERT. The thing we need is to have the latter separable and configurable, because not every piece of software works in a trusted environment. Other than that, constraint assertions, when enabled, are not anyhow different from normal assertions in the sense of action taken. Assertions have configurable breakpoints and deadloops, and DEBUG prints go through a different route in DebugLib that may cause entirely different effects. For example, we halt execution upon printing to DEBUG_ERROR in our DebugLib even in release builds. =To make it clear, currently I plan to add the following interface: #define CONSTRAINT_ASSERT(Expression) \ do { \ if (DebugConstraintAssertEnabled ()) { \ if (!(Expression)) { \ _ASSERT (Expression); \ ANALYZER_UNREACHABLE (); \ } \ } \ } while (FALSE) with DebugConstraintAssertEnabled implemented as (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED | DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED) == DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED) Your suggestion with require macros looks interesting indeed, but I believe it is in fact parallel to this discussion. The change we discuss introduces a new assertion primitive — constraint assertions, while REQUIRE macros are mostly about advanced syntax sugar and higher level assertion primitives on top of existing ones. Perhaps we can have this and make a good use of it, especially given that it brought some practical benefit in Apple, but I would rather discuss this later once constraint assertions are merged into EDK II tree. Best wishes, Vitaly On Sat, Feb 15, 2020 at 03:02, Andrew Fish <afish@apple.com<mailto:afish@apple.com>> wrote: On Feb 14, 2020, at 2:50 PM, Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote: Hi Vitaly, I agree that this proposal makes a lot of sense. We recently added a new assert type called STATIC_ASSERT() for assert conditions that can be tested at build time. A new assert type for checks that can be removed and the API still guarantees that it fails gracefully with a proper return code is a good idea. Given we have STATIC_ASSERT(), how about naming the new macro CONSTRAINT_ASSERT()? We also want the default to be enabled. The current use of bit 0x40 in PcdDebugPropertyMask is always clear, so we want the asserts enabled when 0x40 is clear. We can change the name of the define bit to DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED so bit 0x40 needs to be set in PcdDebugPropertyMask to disable these types of asserts. This approach does require all the DebugLib implementations to be updated with the new DebugConstraintAssertDisabled() API. Mike, If you wanted to be backward compatible you could just use DebugAssertEnabled () but in place of _ASSERT() use _CONSTRAINT_ASSERT #define _ASSERT(Expression) DebugAssert (__FILE__, __LINE__, #Expression) #define _CONSTRAINT_ASSERT(Expression) DebugPrint (DEBUG_ERROR, "ASSERT %a(%d): %a\n",, __FILE__, __LINE__, #Expression) Not as elegant as the non backward compatible change, but I thought I'd throw it out there. There are some ancient Apple C ASSERT macros [AssertMacros.h] that also have the concept of require. Require includes an exception label (goto label). It is like a CONSTRAINT_ASSERT() but with the label. On release builds the DEBUG prints are skipped. So we could do something like: EFI_STATUS Status = EFI_INVALID_PARAMETER; REQUIRE(Arg1 != NULL, ErrorExit); REQUIRE(Arg2 != NULL, ErrorExit); REQUIRE(Arg3 != NULL, ErrorExit); ErrorExit: return Status; There is another form that allows an ACTION (a statement to execute. So you could have: EFI_STATUS Status; REQUIRE_ACTION(Arg1 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); REQUIRE_ACTION(Arg2 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); REQUIRE_ACTION(Arg3 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); ErrorExit: return Status; If you use CONSTRAINT_ASSERT(); if (Arg1 == NULL || Arg2 == NULL || Arg3 == NULL) { CONSTRAINT_ASSERT (Arg1 != NULL); CONSTRAINT_ASSERT (Arg2 != NULL); CONSTRAINT_ASSERT (Arg3 != NULL); return EFI_INVALID_PARAMETER; } I'd note error processing args on entry is the simplest case. In a more complex case when cleanup is required the goto label is more useful. I guess we could argue for less typing and more symmetry and say use ASSERT, CONSTRAINT, and REQUIRE. I guess you could add an ASSERT_ACTION too. The AssertMacros.h versions also support _quiet (skip the print) and _string (add a string to the print) so you end up with: REQUIRE REQUIRE_STRING REQUIRE_QUIET REQUIRE_ACTION REQUIRE_ACTION_STRING REQUIRE_ACTION_QUIET We could also end up with CONSTRAINT CONSTRAINT_STRING CONSTRAINT_QUIET I think the main idea behind _QUIET is you can silence things that are too noisy, and you can easily make noise things show up by removing the _QUIET to debug. I'd thought I throw out the other forms for folks to think about. I'm probably biased as I used to looking at code and seeing things like require_action_string(Arg1 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER, "1st Arg1 check"); Thanks, Andrew Fish PS The old debug macros had 2 versions of CONSTRAINT check and verify. The check version was compiled out on a release build, the verify version always does the check and just skips the DEBUG print. Mike From: vit9696 <vit9696@protonmail.com<mailto:vit9696@protonmail.com>> Sent: Friday, February 14, 2020 9:38 AM To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> Cc: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Gao, Zhichao <zhichao.gao@intel.com<mailto:zhichao.gao@intel.com>>; Marvin Häuser <marvin.haeuser@outlook.com<mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek <lersek@redhat.com<mailto:lersek@redhat.com>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions Michael, Generalising the approach makes good sense to me, but we need to make an obvious distinguishable difference between: - precondition and invariant assertions (i.e. assertions, where function will NOT work if they are violated) - constraint asserts (i.e. assertions, which allow us to spot unintentional behaviour when parsing untrusted data, but which do not break function behaviour). As we want to use this outside of SafeString, I suggest the following: - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED 0x40 bit for PcdDebugPropertyMask instead of PcdAssertOnSafeStringConstraints. - Introduce DebugAssertConstraintEnabled DebugLib function to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. - Introduce ASSERT_CONSTRAINT macro, that will assert only if DebugConstraintAssertEnabled returns true. - Change SafeString ASSERTS in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. - Use ASSERT_CONSTRAINT in other places where necessary. I believe this way lines best with EDK II design. If there are no objections, I can submit the patch in the beginning of next week. Best wishes, Vitaly 14 февр. 2020 г., в 20:00, Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> написал(а): Vitaly, I want to make sure a feature PCD can be used to disable ASSERT() behavior in more than just safe string functions in BaseLib. Can we consider changing the name and description of PcdAssertOnSafeStringConstraints to be more generic, so if we find other lib APIs, the name will make sense? Maybe something like: PcdEnableLibraryAssertChecks? Default is TRUE. Can set to FALSE in DSC file to disable ASSERT() checks. Thanks, Mike From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Vitaly Cheptsov via Groups.Io Sent: Friday, February 14, 2020 3:55 AM To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Gao, Zhichao <zhichao.gao@intel.com<mailto:zhichao.gao@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io> Cc: Marvin Häuser <marvin.haeuser@outlook.com<mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek <lersek@redhat.com<mailto:lersek@redhat.com>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions Replying as per Liming's request for this to be merged into edk2-stable202002. On Mon, Feb 10, 2020 at 14:12, vit9696 <vit9696@protonmail.com<mailto:vit9696@protonmail.com>> wrote: Hello, It has been quite some time since we submitted the patch with so far no negative response. As I mentioned previously, my team will strongly benefit from its landing in EDK II mainline. Since it does not add any regressions and can be viewed as a feature implementation for the rest of EDK II users, I request this to be merged upstream in edk2-stable202002. Best wishes, Vitaly > 27 янв. 2020 г., в 12:47, vit9696 <vit9696@protonmail.com<mailto:vit9696@protonmail.com>> написал(а): > > > Hi Mike, > > Any progress with this? We would really benefit from this landing in the next stable release. > > Best, > Vitaly > >> 8 янв. 2020 г., в 19:35, Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> написал(а): >> >> >> Hi Vitaly, >> >> Thanks for the additional background. I would like >> a couple extra day to review the PCD name and the places >> the PCD might potentially be used. >> >> If we find other APIs where ASSERT() behavior is only >> valuable during dev/debug to quickly identify misuse >> with trusted data and the API provides predicable >> return behavior when ASSERT() is disabled, then I would >> like to have a pattern we can potentially apply to all >> these APIs across all packages. >> >> Thanks, >> >> Mike >> >>> -----Original Message----- >>> From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On >>> Behalf Of Vitaly Cheptsov via Groups.Io >>> Sent: Monday, January 6, 2020 10:44 AM >>> To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> >>> Cc: devel@edk2.groups.io<mailto:devel@edk2.groups.io> >>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>> disable safe string constraint assertions >>> >>> Hi Mike, >>> >>> Yes, the primary use case is for UEFI Applications. We >>> do not want to disable ASSERT’s completely, as >>> assertions that make sense, i.e. the ones signalising >>> about interface misuse, are helpful for debugging. >>> >>> I have already explained in the BZ that basically all >>> safe string constraint assertions make no sense for >>> handling untrusted data. We find this use case very >>> logical, as these functions behave properly with >>> assertions disabled and cover all these error >>> conditions by the return statuses. In such situation is >>> not useful for these functions to assert, as we end up >>> inefficiently reimplementing the logic. I would have >>> liked the approach of discussing the interfaces >>> individually, but I struggle to find any that makes >>> sense from this point of view. >>> >>> AsciiStrToGuid will ASSERT when the length of the >>> passed string is odd. Functions that cannot, ahem, >>> parse, for us are pretty much useless. >>> AsciiStrCatS will ASSERT when the appended string does >>> not fit the buffer. For us this logic makes this >>> function pretty much equivalent to deprecated and thus >>> unavailable AsciiStrCat, except it is also slower. >>> >>> My original suggestion was to remove the assertions >>> entirely, but several people here said that they use >>> them to verify usage errors when handling trusted data. >>> This makes good sense to me, so we suggest to support >>> both cases by introducing a PCD in this patch. >>> >>> Best wishes, >>> Vitaly >>> >>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >>> <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> написал(а): >>>> >>>> >>>> Hi Vitaly, >>>> >>>> Is the use case for UEFI Applications? >>>> >>>> There is a different mechanism to disable all >>> ASSERT() >>>> statements within a UEFI Application. >>>> >>>> If a component is consuming data from an untrusted >>> source, >>>> then that component is required to verify the >>> untrusted >>>> data before passing it to a function that clearly >>> documents >>>> is input requirements. If this approach is followed, >>> then >>>> the BaseLib functions can be used "as is" as long as >>> the >>>> ASSERT() conditions are verified before calling. >>>> >>>> If there are some APIs that currently document their >>> ASSERT() >>>> behavior and we think that ASSERT() behavior is >>> incorrect and >>>> should be handled by an existing error return value, >>> then we >>>> should discuss each of those APIs individually. >>>> >>>> Mike >>>> >>>> >>>>> -----Original Message----- >>>>> From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On >>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>> Sent: Friday, January 3, 2020 9:13 AM >>>>> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> >>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >>> disable >>>>> safe string constraint assertions >>>>> >>>>> REF: >>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >>>>> >>>>> Requesting for merge in edk2-stable202002. >>>>> >>>>> Changes since V1: >>>>> - Enable assertions by default to preserve the >>> original >>>>> behaviour >>>>> - Fix bugzilla reference link >>>>> - Update documentation in BaseLib.h >>>>> >>>>> Vitaly Cheptsov (1): >>>>> MdePkg: Add PCD to disable safe string constraint >>>>> assertions >>>>> >>>>> MdePkg/MdePkg.dec | 6 ++ >>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >>>>> MdePkg/Include/Library/BaseLib.h | 74 >>>>> +++++++++++++------- >>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >>>>> MdePkg/MdePkg.uni | 6 ++ >>>>> 5 files changed, 71 insertions(+), 30 deletions(-) >>>>> >>>>> -- >>>>> 2.21.0 (Apple Git-122.2) >>>>> >>>>> >>>>> -=-=-=-=-=-= >>>>> Groups.io<http://groups.io/> Links: You receive all messages sent to >>> this >>>>> group. >>>>> >>>>> View/Reply Online (#52837): >>>>> https://edk2.groups.io/g/devel/message/52837 >>>>> Mute This Topic: >>> https://groups.io/mt/69401948/1643496 >>>>> Group Owner: devel+owner@edk2.groups.io<mailto:devel+owner@edk2.groups.io> >>>>> Unsubscribe: https://edk2.groups.io/g/devel/unsub >>>>> [michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>] >>>>> -=-=-=-=-=-= >>>> >>> >>> >>> >> > [-- Attachment #2: Type: text/html, Size: 104807 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-15 19:38 ` Michael D Kinney @ 2020-02-16 21:25 ` Andrew Fish 2020-02-17 6:55 ` Vitaly Cheptsov ` (2 more replies) 0 siblings, 3 replies; 28+ messages in thread From: Andrew Fish @ 2020-02-16 21:25 UTC (permalink / raw) To: devel, Mike Kinney Cc: vit9696, Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek [-- Attachment #1: Type: text/plain, Size: 25337 bytes --] Mike, Sorry I don't think I totally understood what felt wrong to me, so I did a bad job explaining my concerns. Also I don't think I was thinking enough in the context of the C11 StdLib. I think my concern is still the global scope of the constraint, even if we tune it per module. For example the DXE Core can interact with PE/COFF images and other data that could have indirectly come from the disk. So conceptually you may want to ASSERT on some constraints and not on others. I think that is why I ratholed on expanding the error handling macros as that was more in the vein of having the caller deal with it, so that is kind of like what you would do pre C11. Also the DebugLib is probably one of the MdePkg Libs that has the most instances floating around, so I think we should change it in a non backwards way very carefully. So after reading up on the C11 implementation of Constraints I think my alternate proposal is for us to add a ConstraintLib modeled after C11 vs. updating the DebugLib. This would solve the 2 things that made me uncomfortable: 1) Extending the DebugLib API; 2) Giving the caller control of the ASSERT behavior. It would still have the down side of breaking builds as the BaseLib would get a new dependency, so we could talk about adding these functions to the DebugLib as the cost of replicating code. C11 defines constraint_handler_t and set_constraint_handler_s as a way for the caller to configure the behavior for bounds checked functions. I think that is the behavior I prefer. So if we are going to make a change that impacts DebugLib compatibility I just want to make sure we have a conversation about all the options. My primary goal is we have the conversation, and if folks don't agree with me that is fine at least we talked about it. What I'm thinking about is as simply exposing an API to control the Constraint handler like C11. This could be done via an ConstrainLib or extending the DebugLib. The basic implementation of the lib would look like: typedef VOID (EFIAPI *CONSTRAINT_HANDLER) ( IN CONST CHAR8 *FileName, IN UINTN LineNumber, IN CONST CHAR8 *Description, IN EFI_STATUS Status ); // Default to AssertConstraintHandler to make it easier to implement Base and XIP libs. // We could have a PCD that also sets the default handler in a Lib Constructor. The default handler is implementation defined in C11. CONSTRAINT_HANDLER gDefaultConstraintHandler = AssertConstraintHandler; CONSTRAINT_HANDLER gActiveConstraintHandler = gDefaultConstraintHandler; BOOLEAN EFIAPI ConstraintAssertEnabled ( VOID ) { return (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & DEBUG_PROPERTY_DEBUG_CONSTRAINT_ENABLED) != 0); } EFI_STATUS EFIAPI SetConstraintHandler ( IN CONSTRAINT_HANDLER Handler ) { if (Handler == NULL) { gActiveConstraintHandler = gDefaultConstraintHandler; } else { gActiveConstraintHandler = Handler; } } VOID AssertConstraintHandler ( IN CONST CHAR8 *FileName, IN UINTN LineNumber, IN CONST CHAR8 *Description, IN EFI_STATUS Status ) { if (ConstraintAssertEnabled ()) { DEBUG ((EFI_D_ERROR, "\Constraint ASSERT (Status = %r): ", Status)); DebugAssert (FileName, LineNumber, Description) } return; } VOID IgnoreConstraintHandler ( IN CONST CHAR8 *FileName, IN UINTN LineNumber, IN CONST CHAR8 *Description, IN EFI_STATUS Status ) { return; } We could add macros for the code in the lib to call: #define CONSTRAINT_CHECK(Expression, Status) \ do { \ if (!(Expression)) { \ gActiveConstraintHandler (__FILE__, __LINE__, Expression, Status); \ return Status; \ } \ } while (FALSE) #define CONSTRAINT_REQUIRE(Expression, Status, Label) \ do { \ if (!(Expression)) { \ gActiveConstraintHandler (__FILE__, __LINE__, Expression, Status); \ goto Label; \ } \ } while (FALSE) As a caller we have now have control: EFI_STATUS Status; CHAR16 Dst[2]; SetConstraintHandler (IgnoreConstraintHandler); Status = StrCpyS (Dst, sizeof (Dst), L"Too Long"); Print (L"Dst =%s (%r)\n", Dst, Status); SetConstraintHandler (AssertConstraintHandler); StrCpyS (Dst, sizeof (Dst), L"Too Long"); Thanks, Andrew Fish PS Since I'm on a crazy idea roll another idea would be to add a MdePkgCompatability.dsc.inc file that could be used to future proof adding dependent libs to existing MdePkg libs. So a platform could include this .DSC and that would give them the default library mapping to keep code compiling. It will only work after other platforms start including it, but after that it would give default mappings for dependent libs. In our above example we could have added this and then existing builds that included MdePkgCompatability.dsc.inc would keep compiling. [LibraryClasses] DebugConstraintLib|MdePkg/Library/DebugConstraintLib/DebugConstraintLib.inf > On Feb 15, 2020, at 11:38 AM, Michael D Kinney <michael.d.kinney@intel.com> wrote: > > Andrew, > > I do not think of this as a globally scoped PCD. It can be set in global scope in DSC. But is can also be set to values based on module type or for specific modules. In the case of the safe string functions, I think there is a desire to disable the constraint asserts when building a UEFI App or UEFI Driver and implement those modules to handle the error return values. Enabling the constraint asserts for PEI Core, DXE Core, SMM/MM Core, PEIM, DXE, SMM/MM modules makes sense to find incorrect input to these functions from modules that can guarantee the inputs would never return an error and catch these as part of dev/debug/validation builds. > > I would not expect disabling on a module by module basis to be common. > > I think the rule for API implementations is to only use CONSTRAINT_ASSERT() for conditions that are also checked and return an error or fail with predicable behavior that allows the system to continue to function. ASSERT() is for conditions that the system can notcontinue. > > Best regards, > > Mike > > From: afish@apple.com <mailto:afish@apple.com> <afish@apple.com <mailto:afish@apple.com>> > Sent: Friday, February 14, 2020 10:27 PM > To: vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> > Cc: devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>; Gao, Liming <liming.gao@intel.com <mailto:liming.gao@intel.com>>; Gao, Zhichao <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>; Marvin Häuser <marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>> > Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions > > Vitaly, > > Sorry after I sent the mail I realized it may come across as me asking you to do work and that was not my intent. > > I will point out though that a non backward compatible change to something as fundamental as the DebugLib is a very big deal. I've got a few different custom implementations that would break with this change as Mike proposed. Given breaking every one's debug lib is such a big deal maybe it is something that we should do as a long term plan vs. some incremental fix. So my intent was to start a conversation about what else we might want to change if we are going to break the world. The only think worse than breaking the world is breaking the world frequently. > > I'm also a little worried that we are taking things that are today locally scoped like SAFE_STRING_CONSTRAINT_CHECK and SAFE_PRINT_CONSTRAINT_CHECK and making them global constructs. I think the way others have dealt with things like this is to make them be DEBUG prints vs. ASSERTs. Also even something as simple as SAFE_STRING_CONSTRAINT_CHECK could be called from code that wants ASSERT and CONSTRAINT_ASSERT behavior. It is not clear to me that the low level code knows the right thing to do in a global sense even if there is a PCD. It almost seems like we should have wrappers for the Safe string functions that implement the behavior you want as a caller. I'm not sure about that, but it seems like it is worth talking about? > > Thanks, > > Andrew Fish > > > On Feb 14, 2020, at 7:31 PM, vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> wrote: > > Hi Andrew, > > While your suggestions look interesting, I am afraid they are not particularly what we want to cover with this discussion at the moment. > > Making assertions go through DEBUG printing functions/macros is what we have to strongly disagree about. Assertions and debug prints are separate things configurable by separate PCDs. We should not mix them. Introducing constraint assertions is a logical step forward in understanding that different software works in different environments. > There are normal, or, as I call them, invariant assertions (e.g. preconditions), for places where the function cannot work unless the assertion is satisfied. This is where we ASSERT. > There are constraint assertions, which signalise that bad data came through the function, even though the function was called from a trusted source. This is where we call CONSTRAINT_ASSERT. > The thing we need is to have the latter separable and configurable, because not every piece of software works in a trusted environment. Other than that, constraint assertions, when enabled, are not anyhow different from normal assertions in the sense of action taken. Assertions have configurable breakpoints and deadloops, and DEBUG prints go through a different route in DebugLib that may cause entirely different effects. For example, we halt execution upon printing to DEBUG_ERROR in our DebugLib even in release builds. > > =To make it clear, currently I plan to add the following interface: > > #define CONSTRAINT_ASSERT(Expression) \ > do { \ > if (DebugConstraintAssertEnabled ()) { \ > if (!(Expression)) { \ > _ASSERT (Expression); \ > ANALYZER_UNREACHABLE (); \ > } \ > } \ > } while (FALSE) > > with DebugConstraintAssertEnabled implemented as > > (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED | DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED) == DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED) > > Your suggestion with require macros looks interesting indeed, but I believe it is in fact parallel to this discussion. The change we discuss introduces a new assertion primitive — constraint assertions, while REQUIRE macros are mostly about advanced syntax sugar and higher level assertion primitives on top of existing ones. Perhaps we can have this and make a good use of it, especially given that it brought some practical benefit in Apple, but I would rather discuss this later once constraint assertions are merged into EDK II tree. > > Best wishes, > Vitaly > > On Sat, Feb 15, 2020 at 03:02, Andrew Fish <afish@apple.com <mailto:afish@apple.com>> wrote: > > > > On Feb 14, 2020, at 2:50 PM, Michael D Kinney <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> wrote: > > Hi Vitaly, > > I agree that this proposal makes a lot of sense. We recently added a new assert type called STATIC_ASSERT() for assert conditions that can be tested at build time. > > A new assert type for checks that can be removed and the API still guarantees that it fails gracefully with a proper return code is a good idea. Given we have STATIC_ASSERT(), how about naming the new macro CONSTRAINT_ASSERT()? > > We also want the default to be enabled. The current use of bit 0x40 in PcdDebugPropertyMask is always clear, so we want the asserts enabled when 0x40 is clear. We can change the name of the define bit to DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED so bit 0x40 needs to be set in PcdDebugPropertyMask to disable these types of asserts. > > This approach does require all the DebugLib implementations to be updated with the new DebugConstraintAssertDisabled() API. > > > Mike, > > If you wanted to be backward compatible you could just use DebugAssertEnabled () but in place of _ASSERT() use _CONSTRAINT_ASSERT > > #define _ASSERT(Expression) DebugAssert (__FILE__, __LINE__, #Expression) > > #define _CONSTRAINT_ASSERT(Expression) DebugPrint (DEBUG_ERROR, "ASSERT %a(%d): %a\n",, __FILE__, __LINE__, #Expression) > > Not as elegant as the non backward compatible change, but I thought I'd throw it out there. > > There are some ancient Apple C ASSERT macros [AssertMacros.h] that also have the concept of require. Require includes an exception label (goto label). It is like a CONSTRAINT_ASSERT() but with the label. On release builds the DEBUG prints are skipped. > > So we could do something like: > > EFI_STATUS Status = EFI_INVALID_PARAMETER; > > REQUIRE(Arg1 != NULL, ErrorExit); > REQUIRE(Arg2 != NULL, ErrorExit); > REQUIRE(Arg3 != NULL, ErrorExit); > > ErrorExit: > return Status; > > There is another form that allows an ACTION (a statement to execute. So you could have: > > EFI_STATUS Status; > > REQUIRE_ACTION(Arg1 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); > REQUIRE_ACTION(Arg2 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); > REQUIRE_ACTION(Arg3 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER); > > ErrorExit: > return Status; > > If you use CONSTRAINT_ASSERT(); > > if (Arg1 == NULL || Arg2 == NULL || Arg3 == NULL) { > CONSTRAINT_ASSERT (Arg1 != NULL); > CONSTRAINT_ASSERT (Arg2 != NULL); > CONSTRAINT_ASSERT (Arg3 != NULL); > return EFI_INVALID_PARAMETER; > } > > I'd note error processing args on entry is the simplest case. In a more complex case when cleanup is required the goto label is more useful. > > I guess we could argue for less typing and more symmetry and say use ASSERT, CONSTRAINT, and REQUIRE. I guess you could add an ASSERT_ACTION too. > > The AssertMacros.h versions also support _quiet (skip the print) and _string (add a string to the print) so you end up with: > REQUIRE > REQUIRE_STRING > REQUIRE_QUIET > REQUIRE_ACTION > REQUIRE_ACTION_STRING > REQUIRE_ACTION_QUIET > > We could also end up with > CONSTRAINT > CONSTRAINT_STRING > CONSTRAINT_QUIET > > I think the main idea behind _QUIET is you can silence things that are too noisy, and you can easily make noise things show up by removing the _QUIET to debug. > > I'd thought I throw out the other forms for folks to think about. I'm probably biased as I used to looking at code and seeing things like require_action_string(Arg1 != NULL, ErrorExit, Status = EFI_INVALID_PARAMETER, "1st Arg1 check"); > > Thanks, > > Andrew Fish > > PS The old debug macros had 2 versions of CONSTRAINT check and verify. The check version was compiled out on a release build, the verify version always does the check and just skips the DEBUG print. > > Mike > > > > From: vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> > Sent: Friday, February 14, 2020 9:38 AM > To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> > Cc: devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Gao, Liming <liming.gao@intel.com <mailto:liming.gao@intel.com>>; Gao, Zhichao <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>; Marvin Häuser <marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>> > Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions > > Michael, > > Generalising the approach makes good sense to me, but we need to make an obvious distinguishable difference between: > - precondition and invariant assertions (i.e. assertions, where function will NOT work if they are violated) > - constraint asserts (i.e. assertions, which allow us to spot unintentional behaviour when parsing untrusted data, but which do not break function behaviour). > > As we want to use this outside of SafeString, I suggest the following: > - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED 0x40 bit for PcdDebugPropertyMask instead of PcdAssertOnSafeStringConstraints. > - Introduce DebugAssertConstraintEnabled DebugLib function to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. > - Introduce ASSERT_CONSTRAINT macro, that will assert only if DebugConstraintAssertEnabled returns true. > - Change SafeString ASSERTS in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. > - Use ASSERT_CONSTRAINT in other places where necessary. > > > I believe this way lines best with EDK II design. If there are no objections, I can submit the patch in the beginning of next week. > > > Best wishes, > Vitaly > > > 14 февр. 2020 г., в 20:00, Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> написал(а): > > Vitaly, > > I want to make sure a feature PCD can be used to disable ASSERT() behavior in more than just safe string functions in BaseLib. > > Can we consider changing the name and description of PcdAssertOnSafeStringConstraints to be more generic, so if we find other lib APIs, the name will make sense? > > Maybe something like: PcdEnableLibraryAssertChecks? Default is TRUE. Can set to FALSE in DSC file to disable ASSERT() checks. > > Thanks, > > Mike > > > > From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On Behalf Of Vitaly Cheptsov via Groups.Io > Sent: Friday, February 14, 2020 3:55 AM > To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>; Gao, Liming <liming.gao@intel.com <mailto:liming.gao@intel.com>>; Gao, Zhichao <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>; devel@edk2.groups.io <mailto:devel@edk2.groups.io> > Cc: Marvin Häuser <marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>> > Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions > > Replying as per Liming's request for this to be merged into edk2-stable202002. > > On Mon, Feb 10, 2020 at 14:12, vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> wrote: > Hello, > > It has been quite some time since we submitted the patch with so far no negative response. As I mentioned previously, my team will strongly benefit from its landing in EDK II mainline. Since it does not add any regressions and can be viewed as a feature implementation for the rest of EDK II users, I request this to be merged upstream in edk2-stable202002. > > Best wishes, > Vitaly > > > 27 янв. 2020 г., в 12:47, vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> написал(а): > > > > > > Hi Mike, > > > > Any progress with this? We would really benefit from this landing in the next stable release. > > > > Best, > > Vitaly > > > >> 8 янв. 2020 г., в 19:35, Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> написал(а): > >> > >> > >> Hi Vitaly, > >> > >> Thanks for the additional background. I would like > >> a couple extra day to review the PCD name and the places > >> the PCD might potentially be used. > >> > >> If we find other APIs where ASSERT() behavior is only > >> valuable during dev/debug to quickly identify misuse > >> with trusted data and the API provides predicable > >> return behavior when ASSERT() is disabled, then I would > >> like to have a pattern we can potentially apply to all > >> these APIs across all packages. > >> > >> Thanks, > >> > >> Mike > >> > >>> -----Original Message----- > >>> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On > >>> Behalf Of Vitaly Cheptsov via Groups.Io > >>> Sent: Monday, January 6, 2020 10:44 AM > >>> To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> > >>> Cc: devel@edk2.groups.io <mailto:devel@edk2.groups.io> > >>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to > >>> disable safe string constraint assertions > >>> > >>> Hi Mike, > >>> > >>> Yes, the primary use case is for UEFI Applications. We > >>> do not want to disable ASSERT’s completely, as > >>> assertions that make sense, i.e. the ones signalising > >>> about interface misuse, are helpful for debugging. > >>> > >>> I have already explained in the BZ that basically all > >>> safe string constraint assertions make no sense for > >>> handling untrusted data. We find this use case very > >>> logical, as these functions behave properly with > >>> assertions disabled and cover all these error > >>> conditions by the return statuses. In such situation is > >>> not useful for these functions to assert, as we end up > >>> inefficiently reimplementing the logic. I would have > >>> liked the approach of discussing the interfaces > >>> individually, but I struggle to find any that makes > >>> sense from this point of view. > >>> > >>> AsciiStrToGuid will ASSERT when the length of the > >>> passed string is odd. Functions that cannot, ahem, > >>> parse, for us are pretty much useless. > >>> AsciiStrCatS will ASSERT when the appended string does > >>> not fit the buffer. For us this logic makes this > >>> function pretty much equivalent to deprecated and thus > >>> unavailable AsciiStrCat, except it is also slower. > >>> > >>> My original suggestion was to remove the assertions > >>> entirely, but several people here said that they use > >>> them to verify usage errors when handling trusted data. > >>> This makes good sense to me, so we suggest to support > >>> both cases by introducing a PCD in this patch. > >>> > >>> Best wishes, > >>> Vitaly > >>> > >>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D > >>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> написал(а): > >>>> > >>>> > >>>> Hi Vitaly, > >>>> > >>>> Is the use case for UEFI Applications? > >>>> > >>>> There is a different mechanism to disable all > >>> ASSERT() > >>>> statements within a UEFI Application. > >>>> > >>>> If a component is consuming data from an untrusted > >>> source, > >>>> then that component is required to verify the > >>> untrusted > >>>> data before passing it to a function that clearly > >>> documents > >>>> is input requirements. If this approach is followed, > >>> then > >>>> the BaseLib functions can be used "as is" as long as > >>> the > >>>> ASSERT() conditions are verified before calling. > >>>> > >>>> If there are some APIs that currently document their > >>> ASSERT() > >>>> behavior and we think that ASSERT() behavior is > >>> incorrect and > >>>> should be handled by an existing error return value, > >>> then we > >>>> should discuss each of those APIs individually. > >>>> > >>>> Mike > >>>> > >>>> > >>>>> -----Original Message----- > >>>>> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On > >>>>> Behalf Of Vitaly Cheptsov via Groups.Io > >>>>> Sent: Friday, January 3, 2020 9:13 AM > >>>>> To: devel@edk2.groups.io <mailto:devel@edk2.groups.io> > >>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to > >>> disable > >>>>> safe string constraint assertions > >>>>> > >>>>> REF: > >>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 <https://bugzilla.tianocore.org/show_bug.cgi?id=2054> > >>>>> > >>>>> Requesting for merge in edk2-stable202002. > >>>>> > >>>>> Changes since V1: > >>>>> - Enable assertions by default to preserve the > >>> original > >>>>> behaviour > >>>>> - Fix bugzilla reference link > >>>>> - Update documentation in BaseLib.h > >>>>> > >>>>> Vitaly Cheptsov (1): > >>>>> MdePkg: Add PCD to disable safe string constraint > >>>>> assertions > >>>>> > >>>>> MdePkg/MdePkg.dec | 6 ++ > >>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- > >>>>> MdePkg/Include/Library/BaseLib.h | 74 > >>>>> +++++++++++++------- > >>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- > >>>>> MdePkg/MdePkg.uni | 6 ++ > >>>>> 5 files changed, 71 insertions(+), 30 deletions(-) > >>>>> > >>>>> -- > >>>>> 2.21.0 (Apple Git-122.2) > >>>>> > >>>>> > >>>>> -=-=-=-=-=-= > >>>>> Groups.io <http://groups.io/> Links: You receive all messages sent to > >>> this > >>>>> group. > >>>>> > >>>>> View/Reply Online (#52837): > >>>>> https://edk2.groups.io/g/devel/message/52837 <https://edk2.groups.io/g/devel/message/52837> > >>>>> Mute This Topic: > >>> https://groups.io/mt/69401948/1643496 <https://groups.io/mt/69401948/1643496> > >>>>> Group Owner: devel+owner@edk2.groups.io <mailto:devel+owner@edk2.groups.io> > >>>>> Unsubscribe: https://edk2.groups.io/g/devel/unsub <https://edk2.groups.io/g/devel/unsub> > >>>>> [michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>] > >>>>> -=-=-=-=-=-= > >>>> > >>> > >>> > >>> > >> > > > > > > > > > > > [-- Attachment #2: Type: text/html, Size: 84332 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-16 21:25 ` Andrew Fish @ 2020-02-17 6:55 ` Vitaly Cheptsov 2020-02-17 8:26 ` Marvin Häuser [not found] ` <15F4232304E080CF.5373@groups.io> 2 siblings, 0 replies; 28+ messages in thread From: Vitaly Cheptsov @ 2020-02-17 6:55 UTC (permalink / raw) To: Andrew Fish, Mike Kinney Cc: devel, Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-16 21:25 ` Andrew Fish 2020-02-17 6:55 ` Vitaly Cheptsov @ 2020-02-17 8:26 ` Marvin Häuser 2020-02-19 23:55 ` Andrew Fish [not found] ` <15F4232304E080CF.5373@groups.io> 2 siblings, 1 reply; 28+ messages in thread From: Marvin Häuser @ 2020-02-17 8:26 UTC (permalink / raw) To: devel@edk2.groups.io, afish@apple.com, Mike Kinney Cc: vit9696, Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek Good day Andrew, First of all, thank you very much for putting this amount of thought into the situation. I definitely agree with the problem you see, but I could also live with Vitaly's proposal. However, I think you are overcomplicating the situation a little. So, we do agree the caller needs control of the behaviour, however your proposal needs "support" from both the caller (choose the handler) and the callee (call the handler) and is neither thread-safe nor event-safe as Vitaly already pointed out. Fixing these problems would only worsen the complexity situation in my opinion. Diverging from both of your concepts entirely and keeping it very simple, what keeps us from simply ASSERTing based on return value? We could introduce a CONSTRAINT_ASSERT or similar, but it would be completely different from Vitaly's proposal. It would be a caller macro to ASSERT based on a pre-defined list of return statuses. To achieve this, we would order all statuses by types (classes). At the moment, I can only think of two: "environmental" and "constraint". For example, "Out Of Resources" would be environmental and "Invalid Parameter" would be constraint. We could define environment statuses explicitly (as it is less likely new environment statuses are introduced) and define constraint statuses as "not environmental" (to silently support new constraint statuses, however of course it is a little error-prone with forwards-compatibility). As a bonus, it forces developers to use truly appropiate error codes and correctly propagate them from callees for this to work. :) This solution would be backwards-compatible in a compilation sense as it can simply be a new macro, however for its possible complexity (switch cases) I might actually prefer a function. But it would not be backwards-compatible with the current functionality, which none of the "caller-based" concepts can be anyway (the caller needs explicit code). However, CONSTRAINT_ASSERT might actually be too specific. Maybe we want something like "ASSERT_RETURN" and have the classes "ASSERT_CONSTRAINT" and "ASSERT_ENVIRONMENT" (bitfield, like with DEBUG). The reason for this is embedded systems where the environment is trusted/known and the firmware is supposed to be well-matched. Think of a SoC with soldered RAM - the firmware can very well make assumption about memory capacity (considering usage across all modules in the worst case) and might actually want ASSERTs on something like "Out Of Resources" because it's supposed to be impossible within the bounds of this specific design. It's possible this would need a new PCD's involvement, for example to tell DxeCore whether to ASSERT on environmental aspects too. There could be an argument-less macro that uses PCD config as base, and an argument-based macro that uses only the parameters. Of course this cannot cover everyone's very specific preferences as it's a per-module switch, but it's the best I can think of right now. Something a bit unrelated now, but it would make this easier. You mentioned PeCoffLib as example, and I think it's a good one, however it has its own return status type within the context[1] which I will most definitely be getting rid of as part of my upcoming (in the far-ish future :) ) RFC. Could we expand RETURN_STATUS to have (very high?) reserved values for caller-defined error codes to have all RETURN_STATUS macros apply to those special return values too? We'd need to define them categorically as "constraint status", but I don't see how a library would declare a new environment status anyway. Regarding MdePkgCompatability.dsc.inc, I think one can override library classes from the inc by declaring them after the !include statement, please correct me if I'm wrong. If so, I strongly agree and think it should be the case for all packages, so one only overrides the defaults when something specific (debugging, performance-optimized versions, ...) is required - easier to read, easier to maintain. The content is needed/there anyway as the libraries are declared in the package's own DSC. It would be nice if you had comments regarding every aspect I just mentioned, it was just something coming to my mind this morning. Thanks for the input so far, it's nice to see some movement now! Best regards, Marvin [1] https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L20-L31 https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L159 Am 16.02.2020 um 22:25 schrieb Andrew Fish via Groups.Io: > Mike, > > Sorry I don't think I totally understood what felt wrong to me, so I did > a bad job explaining my concerns. Also I don't think I was thinking > enough in the context of the C11 StdLib. > > I think my concern is still the global scope of the constraint, even if > we tune it per module. For example the DXE Core can interact with > PE/COFF images and other data that could have indirectly come from the > disk. So conceptually you may want to ASSERT on some constraints and not > on others. I think that is why I ratholed on expanding the error > handling macros as that was more in the vein of having the caller deal > with it, so that is kind of like what you would do pre C11. Also the > DebugLib is probably one of the MdePkg Libs that has the most instances > floating around, so I think we should change it in a non backwards way > very carefully. > > So after reading up on the C11 implementation of Constraints I think my > alternate proposal is for us to add a ConstraintLib modeled after C11 > vs. updating the DebugLib. This would solve the 2 things that made me > uncomfortable: 1) Extending the DebugLib API; 2) Giving the caller > control of the ASSERT behavior. It would still have the down side of > breaking builds as the BaseLib would get a new dependency, so we could > talk about adding these functions to the DebugLib as the cost of > replicating code. > > C11 defines constraint_handler_t and set_constraint_handler_s as a way > for the caller to configure the behavior for bounds checked functions. I > think that is the behavior I prefer. So if we are going to make a change > that impacts DebugLib compatibility I just want to make sure we have a > conversation about all the options. My primary goal is we have the > conversation, and if folks don't agree with me that is fine at least we > talked about it. > > What I'm thinking about is as simply exposing an API to control the > Constraint handler like C11. This could be done via an ConstrainLib or > extending the DebugLib. > > The basic implementation of the lib would look like: > > typedef > VOID > (EFIAPI *CONSTRAINT_HANDLER) ( > IN CONST CHAR8 *FileName, > IN UINTN LineNumber, > IN CONST CHAR8 *Description, > IN EFI_STATUS Status > ); > > > // Default to AssertConstraintHandler to make it easier to implement > Base and XIP libs. > // We could have a PCD that also sets the default handler in a Lib > Constructor. The default handler is implementation defined in C11. > CONSTRAINT_HANDLER gDefaultConstraintHandler = AssertConstraintHandler; > CONSTRAINT_HANDLER gActiveConstraintHandler = gDefaultConstraintHandler; > > BOOLEAN > EFIAPI > ConstraintAssertEnabled ( > VOID > ) > { > return (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & > DEBUG_PROPERTY_DEBUG_CONSTRAINT_ENABLED) != 0); > } > > EFI_STATUS > EFIAPI > SetConstraintHandler ( > IN CONSTRAINT_HANDLER Handler > ) > { > if (Handler == NULL) { > gActiveConstraintHandler = gDefaultConstraintHandler; > } else { > gActiveConstraintHandler = Handler; > } > } > > VOID > AssertConstraintHandler ( > IN CONST CHAR8 *FileName, > IN UINTN LineNumber, > IN CONST CHAR8 *Description, > IN EFI_STATUS Status > ) > { > if (ConstraintAssertEnabled ()) { > DEBUG ((EFI_D_ERROR, "\Constraint ASSERT (Status = %r): ", Status)); > DebugAssert (FileName, LineNumber, Description) > } > > return; > } > > VOID > IgnoreConstraintHandler ( > IN CONST CHAR8 *FileName, > IN UINTN LineNumber, > IN CONST CHAR8 *Description, > IN EFI_STATUS Status > ) > { > return; > } > > We could add macros for the code in the lib to call: > > #define CONSTRAINT_CHECK(Expression, Status) \ > do { \ > if (!(Expression)) { \ > gActiveConstraintHandler (__FILE__, __LINE__, Expression, Status); \ > return Status; \ > } \ > } while (FALSE) > > #define CONSTRAINT_REQUIRE(Expression, Status, Label) \ > do { \ > if (!(Expression)) { \ > gActiveConstraintHandler (__FILE__, __LINE__, Expression, Status); \ > goto Label; \ > } \ > } while (FALSE) > > > As a caller we have now have control: > EFI_STATUS Status; > CHAR16 Dst[2]; > > SetConstraintHandler (IgnoreConstraintHandler); > Status = StrCpyS (Dst, sizeof (Dst), L"Too Long"); > Print (L"Dst =%s (%r)\n", Dst, Status); > > SetConstraintHandler (AssertConstraintHandler); > StrCpyS (Dst, sizeof (Dst), L"Too Long"); > > Thanks, > > Andrew Fish > > PS Since I'm on a crazy idea roll another idea would be to add a > MdePkgCompatability.dsc.inc file that could be used to future proof > adding dependent libs to existing MdePkg libs. So a platform could > include this .DSC and that would give them the default library mapping > to keep code compiling. It will only work after other platforms start > including it, but after that it would give default mappings for > dependent libs. > > In our above example we could have added this and then existing builds > that included MdePkgCompatability.dsc.inc would keep compiling. > > [LibraryClasses] > > DebugConstraintLib|MdePkg/Library/DebugConstraintLib/DebugConstraintLib.inf > > > >> On Feb 15, 2020, at 11:38 AM, Michael D Kinney >> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> wrote: >> >> Andrew, >> I do not think of this as a globally scoped PCD.It can be set in >> global scope in DSC.But is can also be set to values based on module >> type or for specific modules.In the case of the safe string functions, >> I think there is a desire to disable the constraint asserts when >> building a UEFI App or UEFI Driver and implement those modules to >> handle the error return values.Enabling the constraint asserts for PEI >> Core, DXE Core, SMM/MM Core, PEIM, DXE, SMM/MM modules makes sense to >> find incorrect input to these functions from modules that can >> guarantee the inputs would never return an error and catch these as >> part of dev/debug/validation builds. >> I would not expect disabling on a module by module basis to be common. >> I think the rule for API implementations is to only use >> CONSTRAINT_ASSERT() for conditions that are also checked and return an >> error or fail with predicable behavior that allows the system to >> continue to function.ASSERT() is for conditions that the systemcan >> notcontinue. >> Best regards, >> Mike >> *From:*afish@apple.com <mailto:afish@apple.com><afish@apple.com >> <mailto:afish@apple.com>> >> *Sent:*Friday, February 14, 2020 10:27 PM >> *To:*vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> >> *Cc:*devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Kinney, >> Michael D <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>>; Gao, Liming >> <liming.gao@intel.com <mailto:liming.gao@intel.com>>; Gao, Zhichao >> <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>; Marvin Häuser >> <marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com>>; >> Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>> >> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe >> string constraint assertions >> Vitaly, >> Sorry after I sent the mail I realized it may come across as me >> asking you to do work and that was not my intent. >> I will point out though that a non backward compatible change to >> something as fundamental as the DebugLib is a very big deal. I've got >> a few different custom implementations that would break with this >> change as Mike proposed. Given breaking every one's debug lib is such >> a big deal maybe it is something that we should do as a long term plan >> vs. some incremental fix. So my intent was to start a conversation >> about what else we might want to change if we are going to break the >> world. The only think worse than breaking the world is breaking the >> world frequently. >> I'm also a little worried that we are taking things that are today >> locally scoped like SAFE_STRING_CONSTRAINT_CHECK and >> SAFE_PRINT_CONSTRAINT_CHECK and making them global constructs. I think >> the way others have dealt with things like this is to make them be >> DEBUG prints vs. ASSERTs. Also even something as simple as >> SAFE_STRING_CONSTRAINT_CHECK could be called from code that wants >> ASSERT and CONSTRAINT_ASSERT behavior. It is not clear to me that the >> low level code knows the right thing to do in a global sense even if >> there is a PCD. It almost seems like we should have wrappers for the >> Safe string functions that implement the behavior you want as a >> caller. I'm not sure about that, but it seems like it is worth talking >> about? >> Thanks, >> Andrew Fish >> >> >> On Feb 14, 2020, at 7:31 PM, vit9696 <vit9696@protonmail.com >> <mailto:vit9696@protonmail.com>> wrote: >> Hi Andrew, >> While your suggestions look interesting, I am afraid they are not >> particularly what we want to cover with this discussion at the moment. >> Making assertions go through DEBUG printing functions/macros is >> what we have to strongly disagree about. Assertions and debug >> prints are separate things configurable by separate PCDs. We >> should not mix them. Introducing constraint assertions is a >> logical step forward in understanding that different software >> works in different environments. >> >> * There are normal, or, as I call them, invariant >> assertions (e.g. preconditions), for places where the function >> cannot work unless the assertion is satisfied. This is where >> we ASSERT. >> * There are constraint assertions, which signalise that bad data >> came through the function, even though the function was called >> from a trusted source. This is where we call CONSTRAINT_ASSERT. >> >> The thing we need is to have the latter separable >> and configurable, because not every piece of software works in a >> trusted environment. Other than that, constraint assertions, when >> enabled, are not anyhow different from normal assertions in the >> sense of action taken. Assertions have configurable breakpoints >> and deadloops, and DEBUG prints go through a different route in >> DebugLib that may cause entirely different effects. For example, >> we halt execution upon printing to DEBUG_ERROR in our DebugLib >> even in release builds. >> >> =To make it clear, currently I plan to add the following interface: >> #define CONSTRAINT_ASSERT(Expression) \ >> do { \ >> if (DebugConstraintAssertEnabled ()) { \ >> if (!(Expression)) { \ >> _ASSERT (Expression); \ >> ANALYZER_UNREACHABLE (); \ >> } \ >> } \ >> } while (FALSE) >> with DebugConstraintAssertEnabled implemented as >> (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & >> DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED | >> DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED) == >> DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED) >> Your suggestion with require macros looks interesting indeed, but >> I believe it is in fact parallel to this discussion. The change we >> discuss introduces a new assertion primitive — constraint >> assertions, while REQUIRE macros are mostly about advanced syntax >> sugar and higher level assertion primitives on top of existing >> ones. Perhaps we can have this and make a good use of it, >> especially given that it brought some practical benefit in Apple, >> but I would rather discuss this later once constraint assertions >> are merged into EDK II tree. >> Best wishes, >> Vitaly >> On Sat, Feb 15, 2020 at 03:02, Andrew Fish <afish@apple.com >> <mailto:afish@apple.com>> wrote: >> >> >> >> On Feb 14, 2020, at 2:50 PM, Michael D Kinney >> <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>> wrote: >> Hi Vitaly, >> I agree that this proposal makes a lot of sense. We >> recently added a new assert type called STATIC_ASSERT() >> for assert conditions that can be tested at build time. >> A new assert type for checks that can be removed and the >> API still guarantees that it fails gracefully with a >> proper return code is a good idea. Given we have >> STATIC_ASSERT(), how about naming the new macro >> CONSTRAINT_ASSERT()? >> We also want the default to be enabled. The current use of >> bit 0x40 inPcdDebugPropertyMask is always clear, so we >> want the asserts enabled when 0x40 is clear. We can change >> the name of the define bit to >> DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED so bit 0x40 needs >> to be set inPcdDebugPropertyMaskto disable these types of >> asserts. >> This approach does require all theDebugLibimplementations >> to be updated with the newDebugConstraintAssertDisabled() API. >> >> Mike, >> If you wanted to be backward compatible you could just >> use DebugAssertEnabled () but in place of _ASSERT() use >> _CONSTRAINT_ASSERT >> #define _ASSERT(Expression) DebugAssert (__FILE__, __LINE__, >> #Expression) >> #define _CONSTRAINT_ASSERT(Expression) DebugPrint >> (DEBUG_ERROR, "ASSERT %a(%d): %a\n",, __FILE__, __LINE__, >> #Expression) >> Not as elegant as the non backward compatible change, but I >> thought I'd throw it out there. >> There are some ancient Apple C ASSERT macros [AssertMacros.h] >> that also have the concept of require. Require includes an >> exception label (goto label). It is like a CONSTRAINT_ASSERT() >> but with the label. On release builds the DEBUG prints are >> skipped. >> So we could do something like: >> EFI_STATUS Status = EFI_INVALID_PARAMETER; >> REQUIRE(Arg1 != NULL, ErrorExit); >> REQUIRE(Arg2 != NULL, ErrorExit); >> REQUIRE(Arg3 != NULL, ErrorExit); >> ErrorExit: >> return Status; >> There is another form that allows an ACTION (a statement to >> execute. So you could have: >> EFI_STATUS Status; >> REQUIRE_ACTION(Arg1 != NULL, ErrorExit, Status = >> EFI_INVALID_PARAMETER); >> REQUIRE_ACTION(Arg2 != NULL, ErrorExit, Status = >> EFI_INVALID_PARAMETER); >> REQUIRE_ACTION(Arg3 != NULL, ErrorExit, Status = >> EFI_INVALID_PARAMETER); >> ErrorExit: >> return Status; >> If you use CONSTRAINT_ASSERT(); >> if (Arg1 == NULL || Arg2 == NULL || Arg3 == NULL) { >> CONSTRAINT_ASSERT (Arg1 != NULL); >> CONSTRAINT_ASSERT (Arg2 != NULL); >> CONSTRAINT_ASSERT (Arg3 != NULL); >> return EFI_INVALID_PARAMETER; >> } >> I'd note error processing args on entry is the simplest case. >> In a more complex case when cleanup is required the goto >> label is more useful. >> I guess we could argue for less typing and more symmetry and >> say use ASSERT, CONSTRAINT, and REQUIRE. I guess you could add >> an ASSERT_ACTION too. >> The AssertMacros.h versions also support _quiet (skip the >> print) and _string (add a string to the print) so you end up with: >> REQUIRE >> REQUIRE_STRING >> REQUIRE_QUIET >> REQUIRE_ACTION >> REQUIRE_ACTION_STRING >> REQUIRE_ACTION_QUIET >> We could also end up with >> CONSTRAINT >> CONSTRAINT_STRING >> CONSTRAINT_QUIET >> I think the main idea behind _QUIET is you can silence things >> that are too noisy, and you can easily make noise things show >> up by removing the _QUIET to debug. >> I'd thought I throw out the other forms for folks to think >> about. I'm probably biased as I used to looking at code and >> seeing things like require_action_string(Arg1 != NULL, >> ErrorExit, Status = EFI_INVALID_PARAMETER, "1st Arg1 check"); >> Thanks, >> Andrew Fish >> PS The old debug macros had 2 versions of CONSTRAINT check and >> verify. The check version was compiled out on a release build, >> the verify version always does the check and just skips the >> DEBUG print. >> >> Mike >> *From:*vit9696 <vit9696@protonmail.com >> <mailto:vit9696@protonmail.com>> >> *Sent:*Friday, February 14, 2020 9:38 AM >> *To:*Kinney, Michael D <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>> >> *Cc:*devel@edk2.groups.io <mailto:devel@edk2.groups.io>; >> Gao, Liming <liming.gao@intel.com >> <mailto:liming.gao@intel.com>>; Gao, Zhichao >> <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>; >> Marvin Häuser <marvin.haeuser@outlook.com >> <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek >> <lersek@redhat.com <mailto:lersek@redhat.com>> >> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >> disable safe string constraint assertions >> Michael, >> Generalising the approach makes good sense to me, but we >> need to make an obvious distinguishable difference between: >> - precondition and invariant assertions (i.e. assertions, >> where function will NOT work if they are violated) >> - constraint asserts (i.e. assertions, which allow us to >> spot unintentional behaviour when parsing untrusted data, >> but which do not break function behaviour). >> As we want to use this outside of SafeString, I suggest >> the following: >> - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED 0x40 >> bit for PcdDebugPropertyMask instead >> of PcdAssertOnSafeStringConstraints. >> - Introduce DebugAssertConstraintEnabled DebugLib function >> to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. >> - Introduce ASSERT_CONSTRAINT macro, that will assert only >> if DebugConstraintAssertEnabled returns true. >> - Change SafeString ASSERTS >> in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. >> - Use ASSERT_CONSTRAINT in other places where necessary. >> >> I believe this way lines best with EDK II design. If there >> are no objections, I can submit the patch in the beginning >> of next week. >> >> Best wishes, >> Vitaly >> >> 14 февр. 2020 г., в 20:00, Kinney, Michael D >> <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>> написал(а): >> Vitaly, >> I want to make sure a feature PCD can be used to >> disable ASSERT() behavior in more than just safe >> string functions inBaseLib. >> Can we consider changing the name and description >> ofPcdAssertOnSafeStringConstraintsto be more generic, >> so if we find other lib APIs, the name will make sense? >> Maybe something like:PcdEnableLibraryAssertChecks? >> Default is TRUE. Can set to FALSE in DSC file to >> disable ASSERT() checks. >> Thanks, >> Mike >> *From:*devel@edk2.groups.io >> <mailto:devel@edk2.groups.io><devel@edk2.groups.io >> <mailto:devel@edk2.groups.io>>*On Behalf Of*Vitaly >> Cheptsov via Groups.Io >> *Sent:*Friday, February 14, 2020 3:55 AM >> *To:*Kinney, Michael D <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>>; Gao, Liming >> <liming.gao@intel.com <mailto:liming.gao@intel.com>>; >> Gao, Zhichao <zhichao.gao@intel.com >> <mailto:zhichao.gao@intel.com>>;devel@edk2.groups.io >> <mailto:devel@edk2.groups.io> >> *Cc:*Marvin Häuser <marvin.haeuser@outlook.com >> <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek >> <lersek@redhat.com <mailto:lersek@redhat.com>> >> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >> disable safe string constraint assertions >> Replying as per Liming's request for this to be merged >> into edk2-stable202002. >> On Mon, Feb 10, 2020 at 14:12, vit9696 >> <vit9696@protonmail.com >> <mailto:vit9696@protonmail.com>> wrote: >> >> Hello, >> >> It has been quite some time since we submitted the >> patch with so far no negative response. As I >> mentioned previously, my team will strongly >> benefit from its landing in EDK II mainline. Since >> it does not add any regressions and can be viewed >> as a feature implementation for the rest of EDK II >> users, I request this to be merged upstream in >> edk2-stable202002. >> >> Best wishes, >> Vitaly >> >> > 27 янв. 2020 г., в 12:47, vit9696 >> <vit9696@protonmail.com >> <mailto:vit9696@protonmail.com>> написал(а): >> > >> > >> > Hi Mike, >> > >> > Any progress with this? We would really benefit >> from this landing in the next stable release. >> > >> > Best, >> > Vitaly >> > >> >> 8 янв. 2020 г., в 19:35, Kinney, Michael D >> <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>> написал(а): >> >> >> >> >> >> Hi Vitaly, >> >> >> >> Thanks for the additional background. I would like >> >> a couple extra day to review the PCD name and >> the places >> >> the PCD might potentially be used. >> >> >> >> If we find other APIs where ASSERT() behavior >> is only >> >> valuable during dev/debug to quickly identify >> misuse >> >> with trusted data and the API provides predicable >> >> return behavior when ASSERT() is disabled, then >> I would >> >> like to have a pattern we can potentially apply >> to all >> >> these APIs across all packages. >> >> >> >> Thanks, >> >> >> >> Mike >> >> >> >>> -----Original Message----- >> >>> From:devel@edk2.groups.io >> <mailto:devel@edk2.groups.io><devel@edk2.groups.io >> <mailto:devel@edk2.groups.io>> On >> >>> Behalf Of Vitaly Cheptsov via Groups.Io >> >>> Sent: Monday, January 6, 2020 10:44 AM >> >>> To: Kinney, Michael D >> <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>> >> >>> Cc:devel@edk2.groups.io >> <mailto:devel@edk2.groups.io> >> >>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add >> PCD to >> >>> disable safe string constraint assertions >> >>> >> >>> Hi Mike, >> >>> >> >>> Yes, the primary use case is for UEFI >> Applications. We >> >>> do not want to disable ASSERT’s completely, as >> >>> assertions that make sense, i.e. the ones >> signalising >> >>> about interface misuse, are helpful for debugging. >> >>> >> >>> I have already explained in the BZ that >> basically all >> >>> safe string constraint assertions make no >> sense for >> >>> handling untrusted data. We find this use case >> very >> >>> logical, as these functions behave properly with >> >>> assertions disabled and cover all these error >> >>> conditions by the return statuses. In such >> situation is >> >>> not useful for these functions to assert, as >> we end up >> >>> inefficiently reimplementing the logic. I >> would have >> >>> liked the approach of discussing the interfaces >> >>> individually, but I struggle to find any that >> makes >> >>> sense from this point of view. >> >>> >> >>> AsciiStrToGuid will ASSERT when the length of the >> >>> passed string is odd. Functions that cannot, ahem, >> >>> parse, for us are pretty much useless. >> >>> AsciiStrCatS will ASSERT when the appended >> string does >> >>> not fit the buffer. For us this logic makes this >> >>> function pretty much equivalent to deprecated >> and thus >> >>> unavailable AsciiStrCat, except it is also slower. >> >>> >> >>> My original suggestion was to remove the >> assertions >> >>> entirely, but several people here said that >> they use >> >>> them to verify usage errors when handling >> trusted data. >> >>> This makes good sense to me, so we suggest to >> support >> >>> both cases by introducing a PCD in this patch. >> >>> >> >>> Best wishes, >> >>> Vitaly >> >>> >> >>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >> >>> <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>> написал(а): >> >>>> >> >>>> >> >>>> Hi Vitaly, >> >>>> >> >>>> Is the use case for UEFI Applications? >> >>>> >> >>>> There is a different mechanism to disable all >> >>> ASSERT() >> >>>> statements within a UEFI Application. >> >>>> >> >>>> If a component is consuming data from an >> untrusted >> >>> source, >> >>>> then that component is required to verify the >> >>> untrusted >> >>>> data before passing it to a function that clearly >> >>> documents >> >>>> is input requirements. If this approach is >> followed, >> >>> then >> >>>> the BaseLib functions can be used "as is" as >> long as >> >>> the >> >>>> ASSERT() conditions are verified before calling. >> >>>> >> >>>> If there are some APIs that currently >> document their >> >>> ASSERT() >> >>>> behavior and we think that ASSERT() behavior is >> >>> incorrect and >> >>>> should be handled by an existing error return >> value, >> >>> then we >> >>>> should discuss each of those APIs individually. >> >>>> >> >>>> Mike >> >>>> >> >>>> >> >>>>> -----Original Message----- >> >>>>> From:devel@edk2.groups.io >> <mailto:devel@edk2.groups.io><devel@edk2.groups.io >> <mailto:devel@edk2.groups.io>> On >> >>>>> Behalf Of Vitaly Cheptsov via Groups.Io >> >>>>> Sent: Friday, January 3, 2020 9:13 AM >> >>>>> To:devel@edk2.groups.io >> <mailto:devel@edk2.groups.io> >> >>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >> >>> disable >> >>>>> safe string constraint assertions >> >>>>> >> >>>>> REF: >> >>>>>https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >> >>>>> >> >>>>> Requesting for merge in edk2-stable202002. >> >>>>> >> >>>>> Changes since V1: >> >>>>> - Enable assertions by default to preserve the >> >>> original >> >>>>> behaviour >> >>>>> - Fix bugzilla reference link >> >>>>> - Update documentation in BaseLib.h >> >>>>> >> >>>>> Vitaly Cheptsov (1): >> >>>>> MdePkg: Add PCD to disable safe string >> constraint >> >>>>> assertions >> >>>>> >> >>>>> MdePkg/MdePkg.dec | 6 ++ >> >>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >> >>>>> MdePkg/Include/Library/BaseLib.h | 74 >> >>>>> +++++++++++++------- >> >>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >> >>>>> MdePkg/MdePkg.uni | 6 ++ >> >>>>> 5 files changed, 71 insertions(+), 30 >> deletions(-) >> >>>>> >> >>>>> -- >> >>>>> 2.21.0 (Apple Git-122.2) >> >>>>> >> >>>>> >> >>>>> -=-=-=-=-=-= >> >>>>>Groups.io <http://groups.io/>Links: You >> receive all messages sent to >> >>> this >> >>>>> group. >> >>>>> >> >>>>> View/Reply Online (#52837): >> >>>>>https://edk2.groups.io/g/devel/message/52837 >> >>>>> Mute This Topic: >> >>>https://groups.io/mt/69401948/1643496 >> >>>>> Group Owner:devel+owner@edk2.groups.io >> <mailto:devel+owner@edk2.groups.io> >> >>>>> Unsubscribe:https://edk2.groups.io/g/devel/unsub >> >>>>> [michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>] >> >>>>> -=-=-=-=-=-= >> >>>> >> >>> >> >>> >> >>> >> >> >> > >> > > ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-17 8:26 ` Marvin Häuser @ 2020-02-19 23:55 ` Andrew Fish 2020-02-20 10:18 ` Marvin Häuser 0 siblings, 1 reply; 28+ messages in thread From: Andrew Fish @ 2020-02-19 23:55 UTC (permalink / raw) To: devel, mhaeuser Cc: Mike Kinney, vit9696, Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek [-- Attachment #1: Type: text/plain, Size: 40969 bytes --] > On Feb 17, 2020, at 12:26 AM, Marvin Häuser <mhaeuser@outlook.de> wrote: > > Good day Andrew, > > First of all, thank you very much for putting this amount of thought > into the situation. I definitely agree with the problem you see, but I > could also live with Vitaly's proposal. However, I think you are > overcomplicating the situation a little. So, we do agree the caller > needs control of the behaviour, however your proposal needs "support" > from both the caller (choose the handler) and the callee (call the > handler) and is neither thread-safe nor event-safe as Vitaly already > pointed out. Fixing these problems would only worsen the complexity > situation in my opinion. Well EFI does not have Threads but it does have Events. I agree if an Event handler changes the constraint handler it would need to restore it so my proposal is missing an API. CONSTRAINT_HANDLER * GetConstraintHandler ( VOID ) { return gActiveConstraintHandler; } You an always use the standard EFI It is important to remember that all the EFI images (drivers/applications) are statically linked and they carry a unique instance of the Debug (or new Constraint) lib. So in your driver or App is very self contained. Also a lot of the events are phase related callbacks, and since there are no threads your drivers main thread is only really running when your driver, or app, dispatches. A lot of the callbacks are via protocols your driver publishes, but they have strict TPL rules so you can prevent reentrancy in general. So there is a lot more determinism in EFI vs. generic threaded coding. > > Diverging from both of your concepts entirely and keeping it very > simple, what keeps us from simply ASSERTing based on return value? We > could introduce a CONSTRAINT_ASSERT or similar, but it would be > completely different from Vitaly's proposal. It would be a caller macro > to ASSERT based on a pre-defined list of return statuses. > To achieve this, we would order all statuses by types (classes). At the > moment, I can only think of two: "environmental" and "constraint". For > example, "Out Of Resources" would be environmental and "Invalid > Parameter" would be constraint. We could define environment statuses > explicitly (as it is less likely new environment statuses are > introduced) and define constraint statuses as "not environmental" (to > silently support new constraint statuses, however of course it is a > little error-prone with forwards-compatibility). As a bonus, it forces > developers to use truly appropiate error codes and correctly propagate > them from callees for this to work. :) > This solution would be backwards-compatible in a compilation sense as it > can simply be a new macro, however for its possible complexity (switch > cases) I might actually prefer a function. But it would not be > backwards-compatible with the current functionality, which none of the > "caller-based" concepts can be anyway (the caller needs explicit code). > It is an interesting idea to add granularity, but at the end of the knowledge of the correct behavior of the lower level library code really lives in code that calls this. I will admit I can see some value to making RETURN_INVALID_PARAMETER different than RETURN_BUFFER_TOO_SMALL. I kind of went down the C11 path (thanks for mentioning the event issue) but there are other ways to empower the caller. We could let the caller decide the behavior. That implies passing CHECK_DEFAULT (PCD), CHECK_ASSERT, or CHECK_NULL. Then keep backward compatibility. #define StrCpyS(Destination,DestMax,Source) StrCpuSC(Destination, DestMax, Source, CHECK_DEFAULT) But that seems like a lot of thrash to the BaseLib and a lot of new magic to teach developers. I would guess that the most common usage of my library would be to turn Constraint Checking off in the entire driver or app and then handle the errors manually. On driver/app entry do: SetConstraintHandler (IgnoreConstraintHandler); Then handle the errors you care about. Status = StrCpyS (Destination,DestMax,Source); if (EFI_ERROR (Status)) { ASSERT_EFI_ERROR (Status); return Status; } or Status = StrCpyS (Destination,DestMax,Source); if (Status == RETURN_INVALID_PARAMETER) { ASSERT_EFI_ERROR (Status); return Status; } At the end of the day I've code my driver and I know the rules I coded to and I want predictable behavior. I can see a Si vendor developing with ASSERT Constraint off and then when the customer turns it on stuff starts randomly breaking. > However, CONSTRAINT_ASSERT might actually be too specific. Maybe we want > something like "ASSERT_RETURN" and have the classes "ASSERT_CONSTRAINT" > and "ASSERT_ENVIRONMENT" (bitfield, like with DEBUG). The reason for > this is embedded systems where the environment is trusted/known and the > firmware is supposed to be well-matched. Think of a SoC with soldered > RAM - the firmware can very well make assumption about memory capacity > (considering usage across all modules in the worst case) and might > actually want ASSERTs on something like "Out Of Resources" because it's > supposed to be impossible within the bounds of this specific design. > It's possible this would need a new PCD's involvement, for example to > tell DxeCore whether to ASSERT on environmental aspects too. There could > be an argument-less macro that uses PCD config as base, and an > argument-based macro that uses only the parameters. Of course this > cannot cover everyone's very specific preferences as it's a per-module > switch, but it's the best I can think of right now. > > Something a bit unrelated now, but it would make this easier. You > mentioned PeCoffLib as example, and I think it's a good one, however it > has its own return status type within the context[1] which I will most > definitely be getting rid of as part of my upcoming (in the far-ish > future :) ) RFC. Could we expand RETURN_STATUS to have (very high?) > reserved values for caller-defined error codes to have all RETURN_STATUS > macros apply to those special return values too? We'd need to define > them categorically as "constraint status", but I don't see how a library > would declare a new environment status anyway. > > Regarding MdePkgCompatability.dsc.inc, I think one can override library > classes from the inc by declaring them after the !include statement, > please correct me if I'm wrong. If so, I strongly agree and think it > should be the case for all packages, so one only overrides the defaults > when something specific (debugging, performance-optimized versions, ...) > is required - easier to read, easier to maintain. The content is > needed/there anyway as the libraries are declared in the package's own DSC. > Yes it the new .inc DSC file could be overridden by the platform DSC that includes it. I think this is a more general idea than something for this given patch. It is something we could make sure we update every stable tag or so, but I guess someone has to go 1st :). It would make it easier on platforms when they update the edk2 version. > It would be nice if you had comments regarding every aspect I just > mentioned, it was just something coming to my mind this morning. Thanks > for the input so far, it's nice to see some movement now! > Sorry for the delay Thanks, Andrew Fish > Best regards, > Marvin > > [1] > https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L20-L31 <https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L20-L31> > https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L159 <https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L159> > > Am 16.02.2020 um 22:25 schrieb Andrew Fish via Groups.Io: >> Mike, >> >> Sorry I don't think I totally understood what felt wrong to me, so I did >> a bad job explaining my concerns. Also I don't think I was thinking >> enough in the context of the C11 StdLib. >> >> I think my concern is still the global scope of the constraint, even if >> we tune it per module. For example the DXE Core can interact with >> PE/COFF images and other data that could have indirectly come from the >> disk. So conceptually you may want to ASSERT on some constraints and not >> on others. I think that is why I ratholed on expanding the error >> handling macros as that was more in the vein of having the caller deal >> with it, so that is kind of like what you would do pre C11. Also the >> DebugLib is probably one of the MdePkg Libs that has the most instances >> floating around, so I think we should change it in a non backwards way >> very carefully. >> >> So after reading up on the C11 implementation of Constraints I think my >> alternate proposal is for us to add a ConstraintLib modeled after C11 >> vs. updating the DebugLib. This would solve the 2 things that made me >> uncomfortable: 1) Extending the DebugLib API; 2) Giving the caller >> control of the ASSERT behavior. It would still have the down side of >> breaking builds as the BaseLib would get a new dependency, so we could >> talk about adding these functions to the DebugLib as the cost of >> replicating code. >> >> C11 defines constraint_handler_t and set_constraint_handler_s as a way >> for the caller to configure the behavior for bounds checked functions. I >> think that is the behavior I prefer. So if we are going to make a change >> that impacts DebugLib compatibility I just want to make sure we have a >> conversation about all the options. My primary goal is we have the >> conversation, and if folks don't agree with me that is fine at least we >> talked about it. >> >> What I'm thinking about is as simply exposing an API to control the >> Constraint handler like C11. This could be done via an ConstrainLib or >> extending the DebugLib. >> >> The basic implementation of the lib would look like: >> >> typedef >> VOID >> (EFIAPI *CONSTRAINT_HANDLER) ( >> IN CONST CHAR8 *FileName, >> IN UINTN LineNumber, >> IN CONST CHAR8 *Description, >> IN EFI_STATUS Status >> ); >> >> >> // Default to AssertConstraintHandler to make it easier to implement >> Base and XIP libs. >> // We could have a PCD that also sets the default handler in a Lib >> Constructor. The default handler is implementation defined in C11. >> CONSTRAINT_HANDLER gDefaultConstraintHandler = AssertConstraintHandler; >> CONSTRAINT_HANDLER gActiveConstraintHandler = gDefaultConstraintHandler; >> >> BOOLEAN >> EFIAPI >> ConstraintAssertEnabled ( >> VOID >> ) >> { >> return (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & >> DEBUG_PROPERTY_DEBUG_CONSTRAINT_ENABLED) != 0); >> } >> >> EFI_STATUS >> EFIAPI >> SetConstraintHandler ( >> IN CONSTRAINT_HANDLER Handler >> ) >> { >> if (Handler == NULL) { >> gActiveConstraintHandler = gDefaultConstraintHandler; >> } else { >> gActiveConstraintHandler = Handler; >> } >> } >> >> VOID >> AssertConstraintHandler ( >> IN CONST CHAR8 *FileName, >> IN UINTN LineNumber, >> IN CONST CHAR8 *Description, >> IN EFI_STATUS Status >> ) >> { >> if (ConstraintAssertEnabled ()) { >> DEBUG ((EFI_D_ERROR, "\Constraint ASSERT (Status = %r): ", Status)); >> DebugAssert (FileName, LineNumber, Description) >> } >> >> return; >> } >> >> VOID >> IgnoreConstraintHandler ( >> IN CONST CHAR8 *FileName, >> IN UINTN LineNumber, >> IN CONST CHAR8 *Description, >> IN EFI_STATUS Status >> ) >> { >> return; >> } >> >> We could add macros for the code in the lib to call: >> >> #define CONSTRAINT_CHECK(Expression, Status) \ >> do { \ >> if (!(Expression)) { \ >> gActiveConstraintHandler (__FILE__, __LINE__, Expression, Status); \ >> return Status; \ >> } \ >> } while (FALSE) >> >> #define CONSTRAINT_REQUIRE(Expression, Status, Label) \ >> do { \ >> if (!(Expression)) { \ >> gActiveConstraintHandler (__FILE__, __LINE__, Expression, Status); \ >> goto Label; \ >> } \ >> } while (FALSE) >> >> >> As a caller we have now have control: >> EFI_STATUS Status; >> CHAR16 Dst[2]; >> >> SetConstraintHandler (IgnoreConstraintHandler); >> Status = StrCpyS (Dst, sizeof (Dst), L"Too Long"); >> Print (L"Dst =%s (%r)\n", Dst, Status); >> >> SetConstraintHandler (AssertConstraintHandler); >> StrCpyS (Dst, sizeof (Dst), L"Too Long"); >> >> Thanks, >> >> Andrew Fish >> >> PS Since I'm on a crazy idea roll another idea would be to add a >> MdePkgCompatability.dsc.inc file that could be used to future proof >> adding dependent libs to existing MdePkg libs. So a platform could >> include this .DSC and that would give them the default library mapping >> to keep code compiling. It will only work after other platforms start >> including it, but after that it would give default mappings for >> dependent libs. >> >> In our above example we could have added this and then existing builds >> that included MdePkgCompatability.dsc.inc would keep compiling. >> >> [LibraryClasses] >> >> DebugConstraintLib|MdePkg/Library/DebugConstraintLib/DebugConstraintLib.inf >> >> >> >>> On Feb 15, 2020, at 11:38 AM, Michael D Kinney >>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> <mailto:michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>> wrote: >>> >>> Andrew, >>> I do not think of this as a globally scoped PCD.It can be set in >>> global scope in DSC.But is can also be set to values based on module >>> type or for specific modules.In the case of the safe string functions, >>> I think there is a desire to disable the constraint asserts when >>> building a UEFI App or UEFI Driver and implement those modules to >>> handle the error return values.Enabling the constraint asserts for PEI >>> Core, DXE Core, SMM/MM Core, PEIM, DXE, SMM/MM modules makes sense to >>> find incorrect input to these functions from modules that can >>> guarantee the inputs would never return an error and catch these as >>> part of dev/debug/validation builds. >>> I would not expect disabling on a module by module basis to be common. >>> I think the rule for API implementations is to only use >>> CONSTRAINT_ASSERT() for conditions that are also checked and return an >>> error or fail with predicable behavior that allows the system to >>> continue to function.ASSERT() is for conditions that the systemcan >>> notcontinue. >>> Best regards, >>> Mike >>> *From:*afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>><afish@apple.com <mailto:afish@apple.com> >>> <mailto:afish@apple.com <mailto:afish@apple.com>>> >>> *Sent:*Friday, February 14, 2020 10:27 PM >>> *To:*vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com> <mailto:vit9696@protonmail.com <mailto:vit9696@protonmail.com>>> >>> *Cc:*devel@edk2.groups.io <mailto:devel@edk2.groups.io> <mailto:devel@edk2.groups.io <mailto:devel@edk2.groups.io>>; Kinney, >>> Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>> <mailto:michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>>; Gao, Liming >>> <liming.gao@intel.com <mailto:liming.gao@intel.com> <mailto:liming.gao@intel.com <mailto:liming.gao@intel.com>>>; Gao, Zhichao >>> <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com> <mailto:zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>>; Marvin Häuser >>> <marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com> <mailto:marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com>>>; >>> Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com> <mailto:lersek@redhat.com <mailto:lersek@redhat.com>>> >>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe >>> string constraint assertions >>> Vitaly, >>> Sorry after I sent the mail I realized it may come across as me >>> asking you to do work and that was not my intent. >>> I will point out though that a non backward compatible change to >>> something as fundamental as the DebugLib is a very big deal. I've got >>> a few different custom implementations that would break with this >>> change as Mike proposed. Given breaking every one's debug lib is such >>> a big deal maybe it is something that we should do as a long term plan >>> vs. some incremental fix. So my intent was to start a conversation >>> about what else we might want to change if we are going to break the >>> world. The only think worse than breaking the world is breaking the >>> world frequently. >>> I'm also a little worried that we are taking things that are today >>> locally scoped like SAFE_STRING_CONSTRAINT_CHECK and >>> SAFE_PRINT_CONSTRAINT_CHECK and making them global constructs. I think >>> the way others have dealt with things like this is to make them be >>> DEBUG prints vs. ASSERTs. Also even something as simple as >>> SAFE_STRING_CONSTRAINT_CHECK could be called from code that wants >>> ASSERT and CONSTRAINT_ASSERT behavior. It is not clear to me that the >>> low level code knows the right thing to do in a global sense even if >>> there is a PCD. It almost seems like we should have wrappers for the >>> Safe string functions that implement the behavior you want as a >>> caller. I'm not sure about that, but it seems like it is worth talking >>> about? >>> Thanks, >>> Andrew Fish >>> >>> >>> On Feb 14, 2020, at 7:31 PM, vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com> >>> <mailto:vit9696@protonmail.com <mailto:vit9696@protonmail.com>>> wrote: >>> Hi Andrew, >>> While your suggestions look interesting, I am afraid they are not >>> particularly what we want to cover with this discussion at the moment. >>> Making assertions go through DEBUG printing functions/macros is >>> what we have to strongly disagree about. Assertions and debug >>> prints are separate things configurable by separate PCDs. We >>> should not mix them. Introducing constraint assertions is a >>> logical step forward in understanding that different software >>> works in different environments. >>> >>> * There are normal, or, as I call them, invariant >>> assertions (e.g. preconditions), for places where the function >>> cannot work unless the assertion is satisfied. This is where >>> we ASSERT. >>> * There are constraint assertions, which signalise that bad data >>> came through the function, even though the function was called >>> from a trusted source. This is where we call CONSTRAINT_ASSERT. >>> >>> The thing we need is to have the latter separable >>> and configurable, because not every piece of software works in a >>> trusted environment. Other than that, constraint assertions, when >>> enabled, are not anyhow different from normal assertions in the >>> sense of action taken. Assertions have configurable breakpoints >>> and deadloops, and DEBUG prints go through a different route in >>> DebugLib that may cause entirely different effects. For example, >>> we halt execution upon printing to DEBUG_ERROR in our DebugLib >>> even in release builds. >>> >>> =To make it clear, currently I plan to add the following interface: >>> #define CONSTRAINT_ASSERT(Expression) \ >>> do { \ >>> if (DebugConstraintAssertEnabled ()) { \ >>> if (!(Expression)) { \ >>> _ASSERT (Expression); \ >>> ANALYZER_UNREACHABLE (); \ >>> } \ >>> } \ >>> } while (FALSE) >>> with DebugConstraintAssertEnabled implemented as >>> (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & >>> DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED | >>> DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED) == >>> DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED) >>> Your suggestion with require macros looks interesting indeed, but >>> I believe it is in fact parallel to this discussion. The change we >>> discuss introduces a new assertion primitive — constraint >>> assertions, while REQUIRE macros are mostly about advanced syntax >>> sugar and higher level assertion primitives on top of existing >>> ones. Perhaps we can have this and make a good use of it, >>> especially given that it brought some practical benefit in Apple, >>> but I would rather discuss this later once constraint assertions >>> are merged into EDK II tree. >>> Best wishes, >>> Vitaly >>> On Sat, Feb 15, 2020 at 03:02, Andrew Fish <afish@apple.com <mailto:afish@apple.com> >>> <mailto:afish@apple.com <mailto:afish@apple.com>>> wrote: >>> >>> >>> >>> On Feb 14, 2020, at 2:50 PM, Michael D Kinney >>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>> <mailto:michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>> wrote: >>> Hi Vitaly, >>> I agree that this proposal makes a lot of sense. We >>> recently added a new assert type called STATIC_ASSERT() >>> for assert conditions that can be tested at build time. >>> A new assert type for checks that can be removed and the >>> API still guarantees that it fails gracefully with a >>> proper return code is a good idea. Given we have >>> STATIC_ASSERT(), how about naming the new macro >>> CONSTRAINT_ASSERT()? >>> We also want the default to be enabled. The current use of >>> bit 0x40 inPcdDebugPropertyMask is always clear, so we >>> want the asserts enabled when 0x40 is clear. We can change >>> the name of the define bit to >>> DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED so bit 0x40 needs >>> to be set inPcdDebugPropertyMaskto disable these types of >>> asserts. >>> This approach does require all theDebugLibimplementations >>> to be updated with the newDebugConstraintAssertDisabled() API. >>> >>> Mike, >>> If you wanted to be backward compatible you could just >>> use DebugAssertEnabled () but in place of _ASSERT() use >>> _CONSTRAINT_ASSERT >>> #define _ASSERT(Expression) DebugAssert (__FILE__, __LINE__, >>> #Expression) >>> #define _CONSTRAINT_ASSERT(Expression) DebugPrint >>> (DEBUG_ERROR, "ASSERT %a(%d): %a\n",, __FILE__, __LINE__, >>> #Expression) >>> Not as elegant as the non backward compatible change, but I >>> thought I'd throw it out there. >>> There are some ancient Apple C ASSERT macros [AssertMacros.h] >>> that also have the concept of require. Require includes an >>> exception label (goto label). It is like a CONSTRAINT_ASSERT() >>> but with the label. On release builds the DEBUG prints are >>> skipped. >>> So we could do something like: >>> EFI_STATUS Status = EFI_INVALID_PARAMETER; >>> REQUIRE(Arg1 != NULL, ErrorExit); >>> REQUIRE(Arg2 != NULL, ErrorExit); >>> REQUIRE(Arg3 != NULL, ErrorExit); >>> ErrorExit: >>> return Status; >>> There is another form that allows an ACTION (a statement to >>> execute. So you could have: >>> EFI_STATUS Status; >>> REQUIRE_ACTION(Arg1 != NULL, ErrorExit, Status = >>> EFI_INVALID_PARAMETER); >>> REQUIRE_ACTION(Arg2 != NULL, ErrorExit, Status = >>> EFI_INVALID_PARAMETER); >>> REQUIRE_ACTION(Arg3 != NULL, ErrorExit, Status = >>> EFI_INVALID_PARAMETER); >>> ErrorExit: >>> return Status; >>> If you use CONSTRAINT_ASSERT(); >>> if (Arg1 == NULL || Arg2 == NULL || Arg3 == NULL) { >>> CONSTRAINT_ASSERT (Arg1 != NULL); >>> CONSTRAINT_ASSERT (Arg2 != NULL); >>> CONSTRAINT_ASSERT (Arg3 != NULL); >>> return EFI_INVALID_PARAMETER; >>> } >>> I'd note error processing args on entry is the simplest case. >>> In a more complex case when cleanup is required the goto >>> label is more useful. >>> I guess we could argue for less typing and more symmetry and >>> say use ASSERT, CONSTRAINT, and REQUIRE. I guess you could add >>> an ASSERT_ACTION too. >>> The AssertMacros.h versions also support _quiet (skip the >>> print) and _string (add a string to the print) so you end up with: >>> REQUIRE >>> REQUIRE_STRING >>> REQUIRE_QUIET >>> REQUIRE_ACTION >>> REQUIRE_ACTION_STRING >>> REQUIRE_ACTION_QUIET >>> We could also end up with >>> CONSTRAINT >>> CONSTRAINT_STRING >>> CONSTRAINT_QUIET >>> I think the main idea behind _QUIET is you can silence things >>> that are too noisy, and you can easily make noise things show >>> up by removing the _QUIET to debug. >>> I'd thought I throw out the other forms for folks to think >>> about. I'm probably biased as I used to looking at code and >>> seeing things like require_action_string(Arg1 != NULL, >>> ErrorExit, Status = EFI_INVALID_PARAMETER, "1st Arg1 check"); >>> Thanks, >>> Andrew Fish >>> PS The old debug macros had 2 versions of CONSTRAINT check and >>> verify. The check version was compiled out on a release build, >>> the verify version always does the check and just skips the >>> DEBUG print. >>> >>> Mike >>> *From:*vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com> >>> <mailto:vit9696@protonmail.com <mailto:vit9696@protonmail.com>>> >>> *Sent:*Friday, February 14, 2020 9:38 AM >>> *To:*Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>> <mailto:michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>> >>> *Cc:*devel@edk2.groups.io <mailto:devel@edk2.groups.io> <mailto:devel@edk2.groups.io <mailto:devel@edk2.groups.io>>; >>> Gao, Liming <liming.gao@intel.com <mailto:liming.gao@intel.com> >>> <mailto:liming.gao@intel.com <mailto:liming.gao@intel.com>>>; Gao, Zhichao >>> <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com> <mailto:zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>>; >>> Marvin Häuser <marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com> >>> <mailto:marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com>>>; Laszlo Ersek >>> <lersek@redhat.com <mailto:lersek@redhat.com> <mailto:lersek@redhat.com <mailto:lersek@redhat.com>>> >>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>> disable safe string constraint assertions >>> Michael, >>> Generalising the approach makes good sense to me, but we >>> need to make an obvious distinguishable difference between: >>> - precondition and invariant assertions (i.e. assertions, >>> where function will NOT work if they are violated) >>> - constraint asserts (i.e. assertions, which allow us to >>> spot unintentional behaviour when parsing untrusted data, >>> but which do not break function behaviour). >>> As we want to use this outside of SafeString, I suggest >>> the following: >>> - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED 0x40 >>> bit for PcdDebugPropertyMask instead >>> of PcdAssertOnSafeStringConstraints. >>> - Introduce DebugAssertConstraintEnabled DebugLib function >>> to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. >>> - Introduce ASSERT_CONSTRAINT macro, that will assert only >>> if DebugConstraintAssertEnabled returns true. >>> - Change SafeString ASSERTS >>> in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. >>> - Use ASSERT_CONSTRAINT in other places where necessary. >>> >>> I believe this way lines best with EDK II design. If there >>> are no objections, I can submit the patch in the beginning >>> of next week. >>> >>> Best wishes, >>> Vitaly >>> >>> 14 февр. 2020 г., в 20:00, Kinney, Michael D >>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>> <mailto:michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>> написал(а): >>> Vitaly, >>> I want to make sure a feature PCD can be used to >>> disable ASSERT() behavior in more than just safe >>> string functions inBaseLib. >>> Can we consider changing the name and description >>> ofPcdAssertOnSafeStringConstraintsto be more generic, >>> so if we find other lib APIs, the name will make sense? >>> Maybe something like:PcdEnableLibraryAssertChecks? >>> Default is TRUE. Can set to FALSE in DSC file to >>> disable ASSERT() checks. >>> Thanks, >>> Mike >>> *From:*devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>> <mailto:devel@edk2.groups.io <mailto:devel@edk2.groups.io>><devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>> <mailto:devel@edk2.groups.io <mailto:devel@edk2.groups.io>>>*On Behalf Of*Vitaly >>> Cheptsov via Groups.Io >>> *Sent:*Friday, February 14, 2020 3:55 AM >>> *To:*Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>> <mailto:michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>>; Gao, Liming >>> <liming.gao@intel.com <mailto:liming.gao@intel.com> <mailto:liming.gao@intel.com <mailto:liming.gao@intel.com>>>; >>> Gao, Zhichao <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com> >>> <mailto:zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>>;devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>> <mailto:devel@edk2.groups.io <mailto:devel@edk2.groups.io>> >>> *Cc:*Marvin Häuser <marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com> >>> <mailto:marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com>>>; Laszlo Ersek >>> <lersek@redhat.com <mailto:lersek@redhat.com> <mailto:lersek@redhat.com <mailto:lersek@redhat.com>>> >>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>> disable safe string constraint assertions >>> Replying as per Liming's request for this to be merged >>> into edk2-stable202002. >>> On Mon, Feb 10, 2020 at 14:12, vit9696 >>> <vit9696@protonmail.com <mailto:vit9696@protonmail.com> >>> <mailto:vit9696@protonmail.com <mailto:vit9696@protonmail.com>>> wrote: >>> >>> Hello, >>> >>> It has been quite some time since we submitted the >>> patch with so far no negative response. As I >>> mentioned previously, my team will strongly >>> benefit from its landing in EDK II mainline. Since >>> it does not add any regressions and can be viewed >>> as a feature implementation for the rest of EDK II >>> users, I request this to be merged upstream in >>> edk2-stable202002. >>> >>> Best wishes, >>> Vitaly >>> >>>> 27 янв. 2020 г., в 12:47, vit9696 >>> <vit9696@protonmail.com <mailto:vit9696@protonmail.com> >>> <mailto:vit9696@protonmail.com <mailto:vit9696@protonmail.com>>> написал(а): >>>> >>>> >>>> Hi Mike, >>>> >>>> Any progress with this? We would really benefit >>> from this landing in the next stable release. >>>> >>>> Best, >>>> Vitaly >>>> >>>>> 8 янв. 2020 г., в 19:35, Kinney, Michael D >>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>> <mailto:michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>> написал(а): >>>>> >>>>> >>>>> Hi Vitaly, >>>>> >>>>> Thanks for the additional background. I would like >>>>> a couple extra day to review the PCD name and >>> the places >>>>> the PCD might potentially be used. >>>>> >>>>> If we find other APIs where ASSERT() behavior >>> is only >>>>> valuable during dev/debug to quickly identify >>> misuse >>>>> with trusted data and the API provides predicable >>>>> return behavior when ASSERT() is disabled, then >>> I would >>>>> like to have a pattern we can potentially apply >>> to all >>>>> these APIs across all packages. >>>>> >>>>> Thanks, >>>>> >>>>> Mike >>>>> >>>>>> -----Original Message----- >>>>>> From:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>> <mailto:devel@edk2.groups.io <mailto:devel@edk2.groups.io>><devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>> <mailto:devel@edk2.groups.io <mailto:devel@edk2.groups.io>>> On >>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>> Sent: Monday, January 6, 2020 10:44 AM >>>>>> To: Kinney, Michael D >>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>> <mailto:michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>> >>>>>> Cc:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>> <mailto:devel@edk2.groups.io <mailto:devel@edk2.groups.io>> >>>>>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add >>> PCD to >>>>>> disable safe string constraint assertions >>>>>> >>>>>> Hi Mike, >>>>>> >>>>>> Yes, the primary use case is for UEFI >>> Applications. We >>>>>> do not want to disable ASSERT’s completely, as >>>>>> assertions that make sense, i.e. the ones >>> signalising >>>>>> about interface misuse, are helpful for debugging. >>>>>> >>>>>> I have already explained in the BZ that >>> basically all >>>>>> safe string constraint assertions make no >>> sense for >>>>>> handling untrusted data. We find this use case >>> very >>>>>> logical, as these functions behave properly with >>>>>> assertions disabled and cover all these error >>>>>> conditions by the return statuses. In such >>> situation is >>>>>> not useful for these functions to assert, as >>> we end up >>>>>> inefficiently reimplementing the logic. I >>> would have >>>>>> liked the approach of discussing the interfaces >>>>>> individually, but I struggle to find any that >>> makes >>>>>> sense from this point of view. >>>>>> >>>>>> AsciiStrToGuid will ASSERT when the length of the >>>>>> passed string is odd. Functions that cannot, ahem, >>>>>> parse, for us are pretty much useless. >>>>>> AsciiStrCatS will ASSERT when the appended >>> string does >>>>>> not fit the buffer. For us this logic makes this >>>>>> function pretty much equivalent to deprecated >>> and thus >>>>>> unavailable AsciiStrCat, except it is also slower. >>>>>> >>>>>> My original suggestion was to remove the >>> assertions >>>>>> entirely, but several people here said that >>> they use >>>>>> them to verify usage errors when handling >>> trusted data. >>>>>> This makes good sense to me, so we suggest to >>> support >>>>>> both cases by introducing a PCD in this patch. >>>>>> >>>>>> Best wishes, >>>>>> Vitaly >>>>>> >>>>>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >>>>>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>> <mailto:michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>> написал(а): >>>>>>> >>>>>>> >>>>>>> Hi Vitaly, >>>>>>> >>>>>>> Is the use case for UEFI Applications? >>>>>>> >>>>>>> There is a different mechanism to disable all >>>>>> ASSERT() >>>>>>> statements within a UEFI Application. >>>>>>> >>>>>>> If a component is consuming data from an >>> untrusted >>>>>> source, >>>>>>> then that component is required to verify the >>>>>> untrusted >>>>>>> data before passing it to a function that clearly >>>>>> documents >>>>>>> is input requirements. If this approach is >>> followed, >>>>>> then >>>>>>> the BaseLib functions can be used "as is" as >>> long as >>>>>> the >>>>>>> ASSERT() conditions are verified before calling. >>>>>>> >>>>>>> If there are some APIs that currently >>> document their >>>>>> ASSERT() >>>>>>> behavior and we think that ASSERT() behavior is >>>>>> incorrect and >>>>>>> should be handled by an existing error return >>> value, >>>>>> then we >>>>>>> should discuss each of those APIs individually. >>>>>>> >>>>>>> Mike >>>>>>> >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>> <mailto:devel@edk2.groups.io <mailto:devel@edk2.groups.io>><devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>> <mailto:devel@edk2.groups.io <mailto:devel@edk2.groups.io>>> On >>>>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>>>> Sent: Friday, January 3, 2020 9:13 AM >>>>>>>> To:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>> <mailto:devel@edk2.groups.io <mailto:devel@edk2.groups.io>> >>>>>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>>>> disable >>>>>>>> safe string constraint assertions >>>>>>>> >>>>>>>> REF: >>>>>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 <https://bugzilla.tianocore.org/show_bug.cgi?id=2054> >>>>>>>> >>>>>>>> Requesting for merge in edk2-stable202002. >>>>>>>> >>>>>>>> Changes since V1: >>>>>>>> - Enable assertions by default to preserve the >>>>>> original >>>>>>>> behaviour >>>>>>>> - Fix bugzilla reference link >>>>>>>> - Update documentation in BaseLib.h >>>>>>>> >>>>>>>> Vitaly Cheptsov (1): >>>>>>>> MdePkg: Add PCD to disable safe string >>> constraint >>>>>>>> assertions >>>>>>>> >>>>>>>> MdePkg/MdePkg.dec | 6 ++ >>>>>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >>>>>>>> MdePkg/Include/Library/BaseLib.h | 74 >>>>>>>> +++++++++++++------- >>>>>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >>>>>>>> MdePkg/MdePkg.uni | 6 ++ >>>>>>>> 5 files changed, 71 insertions(+), 30 >>> deletions(-) >>>>>>>> >>>>>>>> -- >>>>>>>> 2.21.0 (Apple Git-122.2) >>>>>>>> >>>>>>>> >>>>>>>> -=-=-=-=-=-= >>>>>>>> Groups.io <http://groups.io/> <http://groups.io/ <http://groups.io/>>Links: You >>> receive all messages sent to >>>>>> this >>>>>>>> group. >>>>>>>> >>>>>>>> View/Reply Online (#52837): >>>>>>>> https://edk2.groups.io/g/devel/message/52837 <https://edk2.groups.io/g/devel/message/52837> >>>>>>>> Mute This Topic: >>>>>> https://groups.io/mt/69401948/1643496 <https://groups.io/mt/69401948/1643496> >>>>>>>> Group Owner:devel+owner@edk2.groups.io <mailto:devel+owner@edk2.groups.io> >>> <mailto:devel+owner@edk2.groups.io <mailto:devel+owner@edk2.groups.io>> >>>>>>>> Unsubscribe:https://edk2.groups.io/g/devel/unsub <https://edk2.groups.io/g/devel/unsub> >>>>>>>> [michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>> <mailto:michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>] >>>>>>>> -=-=-=-=-=-= >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >> > > [-- Attachment #2: Type: text/html, Size: 135817 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-19 23:55 ` Andrew Fish @ 2020-02-20 10:18 ` Marvin Häuser 2020-03-03 19:38 ` Marvin Häuser 2020-03-03 20:27 ` Andrew Fish 0 siblings, 2 replies; 28+ messages in thread From: Marvin Häuser @ 2020-02-20 10:18 UTC (permalink / raw) To: Andrew Fish, devel@edk2.groups.io Cc: Mike Kinney, vit9696, Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek Hey Andrew, Thanks once again for your comments, mine are inline. Best regards, Marvin Am 20.02.2020 um 00:55 schrieb Andrew Fish: > > >> On Feb 17, 2020, at 12:26 AM, Marvin Häuser <mhaeuser@outlook.de >> <mailto:mhaeuser@outlook.de>> wrote: >> >> Good day Andrew, >> >> First of all, thank you very much for putting this amount of thought >> into the situation. I definitely agree with the problem you see, but I >> could also live with Vitaly's proposal. However, I think you are >> overcomplicating the situation a little. So, we do agree the caller >> needs control of the behaviour, however your proposal needs "support" >> from both the caller (choose the handler) and the callee (call the >> handler) and is neither thread-safe nor event-safe as Vitaly already >> pointed out. Fixing these problems would only worsen the complexity >> situation in my opinion. > > Well EFI does not have Threads but it does have Events. I agree if an Sorry, we were refering to the UEFI Multi-Processor services, so threads in a physical sense. I think very similar concerns apply regarding these services as with software multithreading in this context, please correct me if I'm wrong. > Event handler changes the constraint handler it would need to restore it > so my proposal is missing an API. > > CONSTRAINT_HANDLER * > GetConstraintHandler ( > VOID > ) > { > return gActiveConstraintHandler; > } > > You an always use the standard EFI > > It is important to remember that all the EFI images > (drivers/applications) are statically linked and they carry a unique > instance of the Debug (or new Constraint) lib. So in your driver or App > is very self contained. Also a lot of the events are phase related > callbacks, and since there are no threads your drivers main thread is > only really running when your driver, or app, dispatches. A lot of the > callbacks are via protocols your driver publishes, but they have strict > TPL rules so you can prevent reentrancy in general. So there is a lot > more determinism in EFI vs. generic threaded coding. This is true, but yet all edge-cases should be covered (event "interruption", actual (processor) interrupts, Multi-Processor execution), so developers do not run into unexpected (seemingly) undeterministic behaviour. However, I agree it is easier to ensure for something like UEFI than for a full-fledged OS of course. > >> >> Diverging from both of your concepts entirely and keeping it very >> simple, what keeps us from simply ASSERTing based on return value? We >> could introduce a CONSTRAINT_ASSERT or similar, but it would be >> completely different from Vitaly's proposal. It would be a caller macro >> to ASSERT based on a pre-defined list of return statuses. >> To achieve this, we would order all statuses by types (classes). At the >> moment, I can only think of two: "environmental" and "constraint". For >> example, "Out Of Resources" would be environmental and "Invalid >> Parameter" would be constraint. We could define environment statuses >> explicitly (as it is less likely new environment statuses are >> introduced) and define constraint statuses as "not environmental" (to >> silently support new constraint statuses, however of course it is a >> little error-prone with forwards-compatibility). As a bonus, it forces >> developers to use truly appropiate error codes and correctly propagate >> them from callees for this to work. :) >> This solution would be backwards-compatible in a compilation sense as it >> can simply be a new macro, however for its possible complexity (switch >> cases) I might actually prefer a function. But it would not be >> backwards-compatible with the current functionality, which none of the >> "caller-based" concepts can be anyway (the caller needs explicit code). >> > > It is an interesting idea to add granularity, but at the end of the > knowledge of the correct behavior of the lower level library code really > lives in code that calls this. I will admit I can see some value to > making RETURN_INVALID_PARAMETER different than RETURN_BUFFER_TOO_SMALL. So far, I prefer this idea as the only requirement is that function contracts are well-designed (i.e. reflect environmental vs constraint errors by using sensical return values). > > I kind of went down the C11 path (thanks for mentioning the event issue) > but there are other ways to empower the caller. It's definitely always a good idea to stick to standards and conventions. It appears my proposal is not conventional at all. However, I also believe there sometimes is a more pragmatic approach to things without imposing significant disadvantages, especially by limiting the scope. To be honest, I'm now interested for which cases the C11 path is advantagous over my approach, except that setting the constraint handler is a one-time thing if it remains consistent throughout execution. I'd imagine for some more complex handling logic I cannot think of an example of (all we really want to do is ASSERT conditionally afterall), and I hope this is not something edk2 would ever run into - keep it simple in my opinion. > > We could let the caller decide the behavior. That implies passing > CHECK_DEFAULT (PCD), CHECK_ASSERT, or CHECK_NULL. > > Then keep backward compatibility. > #define StrCpyS(Destination,DestMax,Source)StrCpuSC(Destination, > DestMax, Source, CHECK_DEFAULT) > > But that seems like a lot of thrash to the BaseLib and a lot of new > magic to teach developers. Fully agreed with the last sentence. That was one of the approaches I mentioned before my proposal, but I included it only for completeness' sake so we have a full overview. This requires prototype adaption for literally every function that may be used in such a way (which scopes beyond just BaseLib), ugly macros and more. Please don't do that. :) > > I would guess that the most common usage of my library would be to turn > Constraint Checking off in the entire driver or app and then handle the > errors manually. I'm afraid that's what most platforms will end up with. Our main concern was we may not allow such constraint violations to ASSERT, that is our main objective. However, if it can be made advantageous to future coding practice, it would be nice to have a decent solution for the caller to have some freedom. More about that below. > > On driver/app entry do: > > SetConstraintHandler (IgnoreConstraintHandler); > > Then handle the errors you care about. > > Status = StrCpyS (Destination,DestMax,Source); > if (EFI_ERROR (Status)) { > ASSERT_EFI_ERROR (Status); > return Status; > } > > or > > Status = StrCpyS (Destination,DestMax,Source); > if (Status == RETURN_INVALID_PARAMETER) { > ASSERT_EFI_ERROR (Status); > return Status; > } > > At the end of the day I've code my driver and I know the rules I coded > to and I want predictable behavior. Yes, agreed. We want predictable behaviour of "not ASSERTing" for untrusted input, because we *know* it is untrusted and the function *can* (and should be able to) handle it - it is simply not a precondition. > > I can see a Si vendor developing with ASSERT Constraint off and then > when the customer turns it on stuff starts randomly breaking. A lot of Si and core (DxeCore, PeiCore, related libraries) code seems to ASSERT (frequently *only* ASSERT with no actual handling, especially for AllocatePool() == NULL) when there are valid error situations possible, so I hope this approach would be a handy tool to satisfy their needs without this absolutely terrible practice. :) I think I understood all your comment individually, but I'm afraid I'm not sure what your suggested solution is right now. You commented semi-positively on my proposal, you did not revoke your C11 path, and you furthermore introduced CHECK_* - this is brainstorming, and I think it's a great thing, but I'm simply not sure what your exact goal or prefered route is right now. My proposal is a bit unconventional and I agree it sounds hacky, but so far I'm not convinced it would be bad practice at all. It boils down to sensefully using the information a function returns. This requires nothing but the function to return sensical information, which every function should anyway. Your proposal is close to the C standard, and that is a good thing first of all. However, with the rest of edk2 not being very compliant, I don't think it's an instant win as in "we just comply to standards" because this choice does not suddenly make porting existing code significantly easier compared to the total lack of standard library support for anything but Shell apps. Pragmatically, I think it solves the same problem equally well, but at a higher cost (ensuring the implementation is safe and, strictly speaking, an ever-so-slight runtime overhead). > >> However, CONSTRAINT_ASSERT might actually be too specific. Maybe we want >> something like "ASSERT_RETURN" and have the classes "ASSERT_CONSTRAINT" >> and "ASSERT_ENVIRONMENT" (bitfield, like with DEBUG). The reason for >> this is embedded systems where the environment is trusted/known and the >> firmware is supposed to be well-matched. Think of a SoC with soldered >> RAM - the firmware can very well make assumption about memory capacity >> (considering usage across all modules in the worst case) and might >> actually want ASSERTs on something like "Out Of Resources" because it's >> supposed to be impossible within the bounds of this specific design. >> It's possible this would need a new PCD's involvement, for example to >> tell DxeCore whether to ASSERT on environmental aspects too. There could >> be an argument-less macro that uses PCD config as base, and an >> argument-based macro that uses only the parameters. Of course this >> cannot cover everyone's very specific preferences as it's a per-module >> switch, but it's the best I can think of right now. >> >> Something a bit unrelated now, but it would make this easier. You >> mentioned PeCoffLib as example, and I think it's a good one, however it >> has its own return status type within the context[1] which I will most >> definitely be getting rid of as part of my upcoming (in the far-ish >> future :) ) RFC. Could we expand RETURN_STATUS to have (very high?) >> reserved values for caller-defined error codes to have all RETURN_STATUS >> macros apply to those special return values too? We'd need to define >> them categorically as "constraint status", but I don't see how a library >> would declare a new environment status anyway. >> >> Regarding MdePkgCompatability.dsc.inc, I think one can override library >> classes from the inc by declaring them after the !include statement, >> please correct me if I'm wrong. If so, I strongly agree and think it >> should be the case for all packages, so one only overrides the defaults >> when something specific (debugging, performance-optimized versions, ...) >> is required - easier to read, easier to maintain. The content is >> needed/there anyway as the libraries are declared in the package's own >> DSC. >> > > Yes it the new .inc DSC file could be overridden by the platform DSC > that includes it. > > I think this is a more general idea than something for this given patch. > It is something we could make sure we update every stable tag or so, but > I guess someone has to go 1st :). It would make it easier on platforms > when they update the edk2 version. Agreed. > > > > >> It would be nice if you had comments regarding every aspect I just >> mentioned, it was just something coming to my mind this morning. Thanks >> for the input so far, it's nice to see some movement now! >> > > Sorry for the delay Sure, thanks for taking time to respond! > > Thanks, > > Andrew Fish > >> Best regards, >> Marvin >> >> [1] >> https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L20-L31 >> https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L159 >> >> Am 16.02.2020 um 22:25 schrieb Andrew Fish via Groups.Io: >>> Mike, >>> >>> Sorry I don't think I totally understood what felt wrong to me, so I did >>> a bad job explaining my concerns. Also I don't think I was thinking >>> enough in the context of the C11 StdLib. >>> >>> I think my concern is still the global scope of the constraint, even if >>> we tune it per module. For example the DXE Core can interact with >>> PE/COFF images and other data that could have indirectly come from the >>> disk. So conceptually you may want to ASSERT on some constraints and not >>> on others. I think that is why I ratholed on expanding the error >>> handling macros as that was more in the vein of having the caller deal >>> with it, so that is kind of like what you would do pre C11. Also the >>> DebugLib is probably one of the MdePkg Libs that has the most instances >>> floating around, so I think we should change it in a non backwards way >>> very carefully. >>> >>> So after reading up on the C11 implementation of Constraints I think my >>> alternate proposal is for us to add a ConstraintLib modeled after C11 >>> vs. updating the DebugLib. This would solve the 2 things that made me >>> uncomfortable: 1) Extending the DebugLib API; 2) Giving the caller >>> control of the ASSERT behavior. It would still have the down side of >>> breaking builds as the BaseLib would get a new dependency, so we could >>> talk about adding these functions to the DebugLib as the cost of >>> replicating code. >>> >>> C11 defines constraint_handler_t and set_constraint_handler_s as a way >>> for the caller to configure the behavior for bounds checked functions. I >>> think that is the behavior I prefer. So if we are going to make a change >>> that impacts DebugLib compatibility I just want to make sure we have a >>> conversation about all the options. My primary goal is we have the >>> conversation, and if folks don't agree with me that is fine at least we >>> talked about it. >>> >>> What I'm thinking about is as simply exposing an API to control the >>> Constraint handler like C11. This could be done via an ConstrainLib or >>> extending the DebugLib. >>> >>> The basic implementation of the lib would look like: >>> >>> typedef >>> VOID >>> (EFIAPI *CONSTRAINT_HANDLER) ( >>> IN CONST CHAR8 *FileName, >>> IN UINTN LineNumber, >>> IN CONST CHAR8 *Description, >>> IN EFI_STATUS Status >>> ); >>> >>> >>> // Default to AssertConstraintHandler to make it easier to implement >>> Base and XIP libs. >>> // We could have a PCD that also sets the default handler in a Lib >>> Constructor. The default handler is implementation defined in C11. >>> CONSTRAINT_HANDLER gDefaultConstraintHandler = AssertConstraintHandler; >>> CONSTRAINT_HANDLER gActiveConstraintHandler = gDefaultConstraintHandler; >>> >>> BOOLEAN >>> EFIAPI >>> ConstraintAssertEnabled ( >>> VOID >>> ) >>> { >>> return (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & >>> DEBUG_PROPERTY_DEBUG_CONSTRAINT_ENABLED) != 0); >>> } >>> >>> EFI_STATUS >>> EFIAPI >>> SetConstraintHandler ( >>> IN CONSTRAINT_HANDLER Handler >>> ) >>> { >>> if (Handler == NULL) { >>> gActiveConstraintHandler = gDefaultConstraintHandler; >>> } else { >>> gActiveConstraintHandler = Handler; >>> } >>> } >>> >>> VOID >>> AssertConstraintHandler ( >>> IN CONST CHAR8 *FileName, >>> IN UINTN LineNumber, >>> IN CONST CHAR8 *Description, >>> IN EFI_STATUS Status >>> ) >>> { >>> if (ConstraintAssertEnabled ()) { >>> DEBUG ((EFI_D_ERROR, "\Constraint ASSERT (Status = %r): ", Status)); >>> DebugAssert (FileName, LineNumber, Description) >>> } >>> >>> return; >>> } >>> >>> VOID >>> IgnoreConstraintHandler ( >>> IN CONST CHAR8 *FileName, >>> IN UINTN LineNumber, >>> IN CONST CHAR8 *Description, >>> IN EFI_STATUS Status >>> ) >>> { >>> return; >>> } >>> >>> We could add macros for the code in the lib to call: >>> >>> #define CONSTRAINT_CHECK(Expression, Status) \ >>> do { \ >>> if (!(Expression)) { \ >>> gActiveConstraintHandler (__FILE__, __LINE__, Expression, Status); \ >>> return Status; \ >>> } \ >>> } while (FALSE) >>> >>> #define CONSTRAINT_REQUIRE(Expression, Status, Label) \ >>> do { \ >>> if (!(Expression)) { \ >>> gActiveConstraintHandler (__FILE__, __LINE__, Expression, Status); \ >>> goto Label; \ >>> } \ >>> } while (FALSE) >>> >>> >>> As a caller we have now have control: >>> EFI_STATUS Status; >>> CHAR16 Dst[2]; >>> >>> SetConstraintHandler (IgnoreConstraintHandler); >>> Status = StrCpyS (Dst, sizeof (Dst), L"Too Long"); >>> Print (L"Dst =%s (%r)\n", Dst, Status); >>> >>> SetConstraintHandler (AssertConstraintHandler); >>> StrCpyS (Dst, sizeof (Dst), L"Too Long"); >>> >>> Thanks, >>> >>> Andrew Fish >>> >>> PS Since I'm on a crazy idea roll another idea would be to add a >>> MdePkgCompatability.dsc.inc file that could be used to future proof >>> adding dependent libs to existing MdePkg libs. So a platform could >>> include this .DSC and that would give them the default library mapping >>> to keep code compiling. It will only work after other platforms start >>> including it, but after that it would give default mappings for >>> dependent libs. >>> >>> In our above example we could have added this and then existing builds >>> that included MdePkgCompatability.dsc.inc would keep compiling. >>> >>> [LibraryClasses] >>> >>> DebugConstraintLib|MdePkg/Library/DebugConstraintLib/DebugConstraintLib.inf >>> >>> >>> >>>> On Feb 15, 2020, at 11:38 AM, Michael D Kinney >>>> <michael.d.kinney@intel.com >>>> <mailto:michael.d.kinney@intel.com><mailto:michael.d.kinney@intel.com>> >>>> wrote: >>>> >>>> Andrew, >>>> I do not think of this as a globally scoped PCD.It can be set in >>>> global scope in DSC.But is can also be set to values based on module >>>> type or for specific modules.In the case of the safe string functions, >>>> I think there is a desire to disable the constraint asserts when >>>> building a UEFI App or UEFI Driver and implement those modules to >>>> handle the error return values.Enabling the constraint asserts for PEI >>>> Core, DXE Core, SMM/MM Core, PEIM, DXE, SMM/MM modules makes sense to >>>> find incorrect input to these functions from modules that can >>>> guarantee the inputs would never return an error and catch these as >>>> part of dev/debug/validation builds. >>>> I would not expect disabling on a module by module basis to be common. >>>> I think the rule for API implementations is to only use >>>> CONSTRAINT_ASSERT() for conditions that are also checked and return an >>>> error or fail with predicable behavior that allows the system to >>>> continue to function.ASSERT() is for conditions that the systemcan >>>> notcontinue. >>>> Best regards, >>>> Mike >>>> *From:*afish@apple.com >>>> <mailto:afish@apple.com><mailto:afish@apple.com><afish@apple.com >>>> <mailto:afish@apple.com> >>>> <mailto:afish@apple.com>> >>>> *Sent:*Friday, February 14, 2020 10:27 PM >>>> *To:*vit9696 <vit9696@protonmail.com >>>> <mailto:vit9696@protonmail.com><mailto:vit9696@protonmail.com>> >>>> *Cc:*devel@edk2.groups.io >>>> <mailto:devel@edk2.groups.io><mailto:devel@edk2.groups.io>; Kinney, >>>> Michael D <michael.d.kinney@intel.com >>>> <mailto:michael.d.kinney@intel.com> >>>> <mailto:michael.d.kinney@intel.com>>; Gao, Liming >>>> <liming.gao@intel.com >>>> <mailto:liming.gao@intel.com><mailto:liming.gao@intel.com>>; Gao, >>>> Zhichao >>>> <zhichao.gao@intel.com >>>> <mailto:zhichao.gao@intel.com><mailto:zhichao.gao@intel.com>>; >>>> Marvin Häuser >>>> <marvin.haeuser@outlook.com >>>> <mailto:marvin.haeuser@outlook.com><mailto:marvin.haeuser@outlook.com>>; >>>> Laszlo Ersek <lersek@redhat.com >>>> <mailto:lersek@redhat.com><mailto:lersek@redhat.com>> >>>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe >>>> string constraint assertions >>>> Vitaly, >>>> Sorry after I sent the mail I realized it may come across as me >>>> asking you to do work and that was not my intent. >>>> I will point out though that a non backward compatible change to >>>> something as fundamental as the DebugLib is a very big deal. I've got >>>> a few different custom implementations that would break with this >>>> change as Mike proposed. Given breaking every one's debug lib is such >>>> a big deal maybe it is something that we should do as a long term plan >>>> vs. some incremental fix. So my intent was to start a conversation >>>> about what else we might want to change if we are going to break the >>>> world. The only think worse than breaking the world is breaking the >>>> world frequently. >>>> I'm also a little worried that we are taking things that are today >>>> locally scoped like SAFE_STRING_CONSTRAINT_CHECK and >>>> SAFE_PRINT_CONSTRAINT_CHECK and making them global constructs. I think >>>> the way others have dealt with things like this is to make them be >>>> DEBUG prints vs. ASSERTs. Also even something as simple as >>>> SAFE_STRING_CONSTRAINT_CHECK could be called from code that wants >>>> ASSERT and CONSTRAINT_ASSERT behavior. It is not clear to me that the >>>> low level code knows the right thing to do in a global sense even if >>>> there is a PCD. It almost seems like we should have wrappers for the >>>> Safe string functions that implement the behavior you want as a >>>> caller. I'm not sure about that, but it seems like it is worth talking >>>> about? >>>> Thanks, >>>> Andrew Fish >>>> >>>> >>>> On Feb 14, 2020, at 7:31 PM, vit9696 <vit9696@protonmail.com >>>> <mailto:vit9696@protonmail.com> >>>> <mailto:vit9696@protonmail.com>> wrote: >>>> Hi Andrew, >>>> While your suggestions look interesting, I am afraid they are not >>>> particularly what we want to cover with this discussion at the >>>> moment. >>>> Making assertions go through DEBUG printing functions/macros is >>>> what we have to strongly disagree about. Assertions and debug >>>> prints are separate things configurable by separate PCDs. We >>>> should not mix them. Introducing constraint assertions is a >>>> logical step forward in understanding that different software >>>> works in different environments. >>>> >>>> * There are normal, or, as I call them, invariant >>>> assertions (e.g. preconditions), for places where the function >>>> cannot work unless the assertion is satisfied. This is where >>>> we ASSERT. >>>> * There are constraint assertions, which signalise that bad data >>>> came through the function, even though the function was called >>>> from a trusted source. This is where we call CONSTRAINT_ASSERT. >>>> >>>> The thing we need is to have the latter separable >>>> and configurable, because not every piece of software works in a >>>> trusted environment. Other than that, constraint assertions, when >>>> enabled, are not anyhow different from normal assertions in the >>>> sense of action taken. Assertions have configurable breakpoints >>>> and deadloops, and DEBUG prints go through a different route in >>>> DebugLib that may cause entirely different effects. For example, >>>> we halt execution upon printing to DEBUG_ERROR in our DebugLib >>>> even in release builds. >>>> >>>> =To make it clear, currently I plan to add the following interface: >>>> #define CONSTRAINT_ASSERT(Expression) \ >>>> do { \ >>>> if (DebugConstraintAssertEnabled ()) { \ >>>> if (!(Expression)) { \ >>>> _ASSERT (Expression); \ >>>> ANALYZER_UNREACHABLE (); \ >>>> } \ >>>> } \ >>>> } while (FALSE) >>>> with DebugConstraintAssertEnabled implemented as >>>> (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & >>>> DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED | >>>> DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED) == >>>> DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED) >>>> Your suggestion with require macros looks interesting indeed, but >>>> I believe it is in fact parallel to this discussion. The change we >>>> discuss introduces a new assertion primitive — constraint >>>> assertions, while REQUIRE macros are mostly about advanced syntax >>>> sugar and higher level assertion primitives on top of existing >>>> ones. Perhaps we can have this and make a good use of it, >>>> especially given that it brought some practical benefit in Apple, >>>> but I would rather discuss this later once constraint assertions >>>> are merged into EDK II tree. >>>> Best wishes, >>>> Vitaly >>>> On Sat, Feb 15, 2020 at 03:02, Andrew Fish <afish@apple.com >>>> <mailto:afish@apple.com> >>>> <mailto:afish@apple.com>> wrote: >>>> >>>> >>>> >>>> On Feb 14, 2020, at 2:50 PM, Michael D Kinney >>>> <michael.d.kinney@intel.com >>>> <mailto:michael.d.kinney@intel.com> >>>> <mailto:michael.d.kinney@intel.com>> wrote: >>>> Hi Vitaly, >>>> I agree that this proposal makes a lot of sense. We >>>> recently added a new assert type called STATIC_ASSERT() >>>> for assert conditions that can be tested at build time. >>>> A new assert type for checks that can be removed and the >>>> API still guarantees that it fails gracefully with a >>>> proper return code is a good idea. Given we have >>>> STATIC_ASSERT(), how about naming the new macro >>>> CONSTRAINT_ASSERT()? >>>> We also want the default to be enabled. The current use of >>>> bit 0x40 inPcdDebugPropertyMask is always clear, so we >>>> want the asserts enabled when 0x40 is clear. We can change >>>> the name of the define bit to >>>> DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED so bit 0x40 needs >>>> to be set inPcdDebugPropertyMaskto disable these types of >>>> asserts. >>>> This approach does require all theDebugLibimplementations >>>> to be updated with the newDebugConstraintAssertDisabled() >>>> API. >>>> >>>> Mike, >>>> If you wanted to be backward compatible you could just >>>> use DebugAssertEnabled () but in place of _ASSERT() use >>>> _CONSTRAINT_ASSERT >>>> #define _ASSERT(Expression) DebugAssert (__FILE__, __LINE__, >>>> #Expression) >>>> #define _CONSTRAINT_ASSERT(Expression) DebugPrint >>>> (DEBUG_ERROR, "ASSERT %a(%d): %a\n",, __FILE__, __LINE__, >>>> #Expression) >>>> Not as elegant as the non backward compatible change, but I >>>> thought I'd throw it out there. >>>> There are some ancient Apple C ASSERT macros [AssertMacros.h] >>>> that also have the concept of require. Require includes an >>>> exception label (goto label). It is like a CONSTRAINT_ASSERT() >>>> but with the label. On release builds the DEBUG prints are >>>> skipped. >>>> So we could do something like: >>>> EFI_STATUS Status = EFI_INVALID_PARAMETER; >>>> REQUIRE(Arg1 != NULL, ErrorExit); >>>> REQUIRE(Arg2 != NULL, ErrorExit); >>>> REQUIRE(Arg3 != NULL, ErrorExit); >>>> ErrorExit: >>>> return Status; >>>> There is another form that allows an ACTION (a statement to >>>> execute. So you could have: >>>> EFI_STATUS Status; >>>> REQUIRE_ACTION(Arg1 != NULL, ErrorExit, Status = >>>> EFI_INVALID_PARAMETER); >>>> REQUIRE_ACTION(Arg2 != NULL, ErrorExit, Status = >>>> EFI_INVALID_PARAMETER); >>>> REQUIRE_ACTION(Arg3 != NULL, ErrorExit, Status = >>>> EFI_INVALID_PARAMETER); >>>> ErrorExit: >>>> return Status; >>>> If you use CONSTRAINT_ASSERT(); >>>> if (Arg1 == NULL || Arg2 == NULL || Arg3 == NULL) { >>>> CONSTRAINT_ASSERT (Arg1 != NULL); >>>> CONSTRAINT_ASSERT (Arg2 != NULL); >>>> CONSTRAINT_ASSERT (Arg3 != NULL); >>>> return EFI_INVALID_PARAMETER; >>>> } >>>> I'd note error processing args on entry is the simplest case. >>>> In a more complex case when cleanup is required the goto >>>> label is more useful. >>>> I guess we could argue for less typing and more symmetry and >>>> say use ASSERT, CONSTRAINT, and REQUIRE. I guess you could add >>>> an ASSERT_ACTION too. >>>> The AssertMacros.h versions also support _quiet (skip the >>>> print) and _string (add a string to the print) so you end up >>>> with: >>>> REQUIRE >>>> REQUIRE_STRING >>>> REQUIRE_QUIET >>>> REQUIRE_ACTION >>>> REQUIRE_ACTION_STRING >>>> REQUIRE_ACTION_QUIET >>>> We could also end up with >>>> CONSTRAINT >>>> CONSTRAINT_STRING >>>> CONSTRAINT_QUIET >>>> I think the main idea behind _QUIET is you can silence things >>>> that are too noisy, and you can easily make noise things show >>>> up by removing the _QUIET to debug. >>>> I'd thought I throw out the other forms for folks to think >>>> about. I'm probably biased as I used to looking at code and >>>> seeing things like require_action_string(Arg1 != NULL, >>>> ErrorExit, Status = EFI_INVALID_PARAMETER, "1st Arg1 check"); >>>> Thanks, >>>> Andrew Fish >>>> PS The old debug macros had 2 versions of CONSTRAINT check and >>>> verify. The check version was compiled out on a release build, >>>> the verify version always does the check and just skips the >>>> DEBUG print. >>>> >>>> Mike >>>> *From:*vit9696 <vit9696@protonmail.com >>>> <mailto:vit9696@protonmail.com> >>>> <mailto:vit9696@protonmail.com>> >>>> *Sent:*Friday, February 14, 2020 9:38 AM >>>> *To:*Kinney, Michael D <michael.d.kinney@intel.com >>>> <mailto:michael.d.kinney@intel.com> >>>> <mailto:michael.d.kinney@intel.com>> >>>> *Cc:*devel@edk2.groups.io >>>> <mailto:devel@edk2.groups.io><mailto:devel@edk2.groups.io>; >>>> Gao, Liming <liming.gao@intel.com >>>> <mailto:liming.gao@intel.com> >>>> <mailto:liming.gao@intel.com>>; Gao, Zhichao >>>> <zhichao.gao@intel.com >>>> <mailto:zhichao.gao@intel.com><mailto:zhichao.gao@intel.com>>; >>>> Marvin Häuser <marvin.haeuser@outlook.com >>>> <mailto:marvin.haeuser@outlook.com> >>>> <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek >>>> <lersek@redhat.com >>>> <mailto:lersek@redhat.com><mailto:lersek@redhat.com>> >>>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>> disable safe string constraint assertions >>>> Michael, >>>> Generalising the approach makes good sense to me, but we >>>> need to make an obvious distinguishable difference between: >>>> - precondition and invariant assertions (i.e. assertions, >>>> where function will NOT work if they are violated) >>>> - constraint asserts (i.e. assertions, which allow us to >>>> spot unintentional behaviour when parsing untrusted data, >>>> but which do not break function behaviour). >>>> As we want to use this outside of SafeString, I suggest >>>> the following: >>>> - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED 0x40 >>>> bit for PcdDebugPropertyMask instead >>>> of PcdAssertOnSafeStringConstraints. >>>> - Introduce DebugAssertConstraintEnabled DebugLib function >>>> to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. >>>> - Introduce ASSERT_CONSTRAINT macro, that will assert only >>>> if DebugConstraintAssertEnabled returns true. >>>> - Change SafeString ASSERTS >>>> in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. >>>> - Use ASSERT_CONSTRAINT in other places where necessary. >>>> >>>> I believe this way lines best with EDK II design. If there >>>> are no objections, I can submit the patch in the beginning >>>> of next week. >>>> >>>> Best wishes, >>>> Vitaly >>>> >>>> 14 февр. 2020 г., в 20:00, Kinney, Michael D >>>> <michael.d.kinney@intel.com >>>> <mailto:michael.d.kinney@intel.com> >>>> <mailto:michael.d.kinney@intel.com>> написал(а): >>>> Vitaly, >>>> I want to make sure a feature PCD can be used to >>>> disable ASSERT() behavior in more than just safe >>>> string functions inBaseLib. >>>> Can we consider changing the name and description >>>> ofPcdAssertOnSafeStringConstraintsto be more generic, >>>> so if we find other lib APIs, the name will make sense? >>>> Maybe something like:PcdEnableLibraryAssertChecks? >>>> Default is TRUE. Can set to FALSE in DSC file to >>>> disable ASSERT() checks. >>>> Thanks, >>>> Mike >>>> *From:*devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>> <mailto:devel@edk2.groups.io><devel@edk2.groups.io >>>> <mailto:devel@edk2.groups.io> >>>> <mailto:devel@edk2.groups.io>>*On Behalf Of*Vitaly >>>> Cheptsov via Groups.Io >>>> *Sent:*Friday, February 14, 2020 3:55 AM >>>> *To:*Kinney, Michael D <michael.d.kinney@intel.com >>>> <mailto:michael.d.kinney@intel.com> >>>> <mailto:michael.d.kinney@intel.com>>; Gao, Liming >>>> <liming.gao@intel.com >>>> <mailto:liming.gao@intel.com><mailto:liming.gao@intel.com>>; >>>> Gao, Zhichao <zhichao.gao@intel.com >>>> <mailto:zhichao.gao@intel.com> >>>> <mailto:zhichao.gao@intel.com>>;devel@edk2.groups.io >>>> <mailto:devel@edk2.groups.io> >>>> <mailto:devel@edk2.groups.io> >>>> *Cc:*Marvin Häuser <marvin.haeuser@outlook.com >>>> <mailto:marvin.haeuser@outlook.com> >>>> <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek >>>> <lersek@redhat.com >>>> <mailto:lersek@redhat.com><mailto:lersek@redhat.com>> >>>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>> disable safe string constraint assertions >>>> Replying as per Liming's request for this to be merged >>>> into edk2-stable202002. >>>> On Mon, Feb 10, 2020 at 14:12, vit9696 >>>> <vit9696@protonmail.com <mailto:vit9696@protonmail.com> >>>> <mailto:vit9696@protonmail.com>> wrote: >>>> >>>> Hello, >>>> >>>> It has been quite some time since we submitted the >>>> patch with so far no negative response. As I >>>> mentioned previously, my team will strongly >>>> benefit from its landing in EDK II mainline. Since >>>> it does not add any regressions and can be viewed >>>> as a feature implementation for the rest of EDK II >>>> users, I request this to be merged upstream in >>>> edk2-stable202002. >>>> >>>> Best wishes, >>>> Vitaly >>>> >>>>> 27 янв. 2020 г., в 12:47, vit9696 >>>> <vit9696@protonmail.com >>>> <mailto:vit9696@protonmail.com> >>>> <mailto:vit9696@protonmail.com>> написал(а): >>>>> >>>>> >>>>> Hi Mike, >>>>> >>>>> Any progress with this? We would really benefit >>>> from this landing in the next stable release. >>>>> >>>>> Best, >>>>> Vitaly >>>>> >>>>>> 8 янв. 2020 г., в 19:35, Kinney, Michael D >>>> <michael.d.kinney@intel.com >>>> <mailto:michael.d.kinney@intel.com> >>>> <mailto:michael.d.kinney@intel.com>> написал(а): >>>>>> >>>>>> >>>>>> Hi Vitaly, >>>>>> >>>>>> Thanks for the additional background. I would like >>>>>> a couple extra day to review the PCD name and >>>> the places >>>>>> the PCD might potentially be used. >>>>>> >>>>>> If we find other APIs where ASSERT() behavior >>>> is only >>>>>> valuable during dev/debug to quickly identify >>>> misuse >>>>>> with trusted data and the API provides predicable >>>>>> return behavior when ASSERT() is disabled, then >>>> I would >>>>>> like to have a pattern we can potentially apply >>>> to all >>>>>> these APIs across all packages. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Mike >>>>>> >>>>>>> -----Original Message----- >>>>>>> From:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>> <mailto:devel@edk2.groups.io><devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>> <mailto:devel@edk2.groups.io>> On >>>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>>> Sent: Monday, January 6, 2020 10:44 AM >>>>>>> To: Kinney, Michael D >>>> <michael.d.kinney@intel.com >>>> <mailto:michael.d.kinney@intel.com> >>>> <mailto:michael.d.kinney@intel.com>> >>>>>>> Cc:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>> <mailto:devel@edk2.groups.io> >>>>>>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add >>>> PCD to >>>>>>> disable safe string constraint assertions >>>>>>> >>>>>>> Hi Mike, >>>>>>> >>>>>>> Yes, the primary use case is for UEFI >>>> Applications. We >>>>>>> do not want to disable ASSERT’s completely, as >>>>>>> assertions that make sense, i.e. the ones >>>> signalising >>>>>>> about interface misuse, are helpful for debugging. >>>>>>> >>>>>>> I have already explained in the BZ that >>>> basically all >>>>>>> safe string constraint assertions make no >>>> sense for >>>>>>> handling untrusted data. We find this use case >>>> very >>>>>>> logical, as these functions behave properly with >>>>>>> assertions disabled and cover all these error >>>>>>> conditions by the return statuses. In such >>>> situation is >>>>>>> not useful for these functions to assert, as >>>> we end up >>>>>>> inefficiently reimplementing the logic. I >>>> would have >>>>>>> liked the approach of discussing the interfaces >>>>>>> individually, but I struggle to find any that >>>> makes >>>>>>> sense from this point of view. >>>>>>> >>>>>>> AsciiStrToGuid will ASSERT when the length of the >>>>>>> passed string is odd. Functions that cannot, ahem, >>>>>>> parse, for us are pretty much useless. >>>>>>> AsciiStrCatS will ASSERT when the appended >>>> string does >>>>>>> not fit the buffer. For us this logic makes this >>>>>>> function pretty much equivalent to deprecated >>>> and thus >>>>>>> unavailable AsciiStrCat, except it is also slower. >>>>>>> >>>>>>> My original suggestion was to remove the >>>> assertions >>>>>>> entirely, but several people here said that >>>> they use >>>>>>> them to verify usage errors when handling >>>> trusted data. >>>>>>> This makes good sense to me, so we suggest to >>>> support >>>>>>> both cases by introducing a PCD in this patch. >>>>>>> >>>>>>> Best wishes, >>>>>>> Vitaly >>>>>>> >>>>>>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >>>>>>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>>> <mailto:michael.d.kinney@intel.com>> написал(а): >>>>>>>> >>>>>>>> >>>>>>>> Hi Vitaly, >>>>>>>> >>>>>>>> Is the use case for UEFI Applications? >>>>>>>> >>>>>>>> There is a different mechanism to disable all >>>>>>> ASSERT() >>>>>>>> statements within a UEFI Application. >>>>>>>> >>>>>>>> If a component is consuming data from an >>>> untrusted >>>>>>> source, >>>>>>>> then that component is required to verify the >>>>>>> untrusted >>>>>>>> data before passing it to a function that clearly >>>>>>> documents >>>>>>>> is input requirements. If this approach is >>>> followed, >>>>>>> then >>>>>>>> the BaseLib functions can be used "as is" as >>>> long as >>>>>>> the >>>>>>>> ASSERT() conditions are verified before calling. >>>>>>>> >>>>>>>> If there are some APIs that currently >>>> document their >>>>>>> ASSERT() >>>>>>>> behavior and we think that ASSERT() behavior is >>>>>>> incorrect and >>>>>>>> should be handled by an existing error return >>>> value, >>>>>>> then we >>>>>>>> should discuss each of those APIs individually. >>>>>>>> >>>>>>>> Mike >>>>>>>> >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>> <mailto:devel@edk2.groups.io><devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>> <mailto:devel@edk2.groups.io>> On >>>>>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>>>>> Sent: Friday, January 3, 2020 9:13 AM >>>>>>>>> To:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>> <mailto:devel@edk2.groups.io> >>>>>>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>>>>> disable >>>>>>>>> safe string constraint assertions >>>>>>>>> >>>>>>>>> REF: >>>>>>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >>>>>>>>> >>>>>>>>> Requesting for merge in edk2-stable202002. >>>>>>>>> >>>>>>>>> Changes since V1: >>>>>>>>> - Enable assertions by default to preserve the >>>>>>> original >>>>>>>>> behaviour >>>>>>>>> - Fix bugzilla reference link >>>>>>>>> - Update documentation in BaseLib.h >>>>>>>>> >>>>>>>>> Vitaly Cheptsov (1): >>>>>>>>> MdePkg: Add PCD to disable safe string >>>> constraint >>>>>>>>> assertions >>>>>>>>> >>>>>>>>> MdePkg/MdePkg.dec | 6 ++ >>>>>>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >>>>>>>>> MdePkg/Include/Library/BaseLib.h | 74 >>>>>>>>> +++++++++++++------- >>>>>>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >>>>>>>>> MdePkg/MdePkg.uni | 6 ++ >>>>>>>>> 5 files changed, 71 insertions(+), 30 >>>> deletions(-) >>>>>>>>> >>>>>>>>> -- >>>>>>>>> 2.21.0 (Apple Git-122.2) >>>>>>>>> >>>>>>>>> >>>>>>>>> -=-=-=-=-=-= >>>>>>>>> Groups.io <http://groups.io/><http://groups.io/>Links: You >>>> receive all messages sent to >>>>>>> this >>>>>>>>> group. >>>>>>>>> >>>>>>>>> View/Reply Online (#52837): >>>>>>>>> https://edk2.groups.io/g/devel/message/52837 >>>>>>>>> Mute This Topic: >>>>>>> https://groups.io/mt/69401948/1643496 >>>>>>>>> Group Owner:devel+owner@edk2.groups.io >>>>>>>>> <mailto:devel+owner@edk2.groups.io> >>>> <mailto:devel+owner@edk2.groups.io> >>>>>>>>> Unsubscribe:https://edk2.groups.io/g/devel/unsub >>>>>>>>> [michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>>> <mailto:michael.d.kinney@intel.com>] >>>>>>>>> -=-=-=-=-=-= >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >> >> > ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-20 10:18 ` Marvin Häuser @ 2020-03-03 19:38 ` Marvin Häuser 2020-03-04 0:19 ` Liming Gao 2020-03-03 20:27 ` Andrew Fish 1 sibling, 1 reply; 28+ messages in thread From: Marvin Häuser @ 2020-03-03 19:38 UTC (permalink / raw) To: Andrew Fish, devel@edk2.groups.io Cc: Mike Kinney, vit9696, Gao, Liming, Gao, Zhichao, Laszlo Ersek Good day, Unfortunately, this discussion has died again. This is fine by me, I'm also fine with the already proposed patch and mostly shared different approaches based on the vastly diverging feedback, but it works for us either way. Can the patch please be reviewed and accepted for the next stable tag? Thanks! Best regards, Marvin Am 20.02.2020 um 11:18 schrieb Marvin Häuser: > Hey Andrew, > > Thanks once again for your comments, mine are inline. > > Best regards, > Marvin > > Am 20.02.2020 um 00:55 schrieb Andrew Fish: >> >> >>> On Feb 17, 2020, at 12:26 AM, Marvin Häuser <mhaeuser@outlook.de >>> <mailto:mhaeuser@outlook.de>> wrote: >>> >>> Good day Andrew, >>> >>> First of all, thank you very much for putting this amount of thought >>> into the situation. I definitely agree with the problem you see, but I >>> could also live with Vitaly's proposal. However, I think you are >>> overcomplicating the situation a little. So, we do agree the caller >>> needs control of the behaviour, however your proposal needs "support" >>> from both the caller (choose the handler) and the callee (call the >>> handler) and is neither thread-safe nor event-safe as Vitaly already >>> pointed out. Fixing these problems would only worsen the complexity >>> situation in my opinion. >> >> Well EFI does not have Threads but it does have Events. I agree if an > > Sorry, we were refering to the UEFI Multi-Processor services, so threads > in a physical sense. I think very similar concerns apply regarding these > services as with software multithreading in this context, please correct > me if I'm wrong. > >> Event handler changes the constraint handler it would need to restore it >> so my proposal is missing an API. >> >> CONSTRAINT_HANDLER * >> GetConstraintHandler ( >> VOID >> ) >> { >> return gActiveConstraintHandler; >> } >> >> You an always use the standard EFI >> >> It is important to remember that all the EFI images >> (drivers/applications) are statically linked and they carry a unique >> instance of the Debug (or new Constraint) lib. So in your driver or App >> is very self contained. Also a lot of the events are phase related >> callbacks, and since there are no threads your drivers main thread is >> only really running when your driver, or app, dispatches. A lot of the >> callbacks are via protocols your driver publishes, but they have strict >> TPL rules so you can prevent reentrancy in general. So there is a lot >> more determinism in EFI vs. generic threaded coding. > > This is true, but yet all edge-cases should be covered (event > "interruption", actual (processor) interrupts, Multi-Processor > execution), so developers do not run into unexpected (seemingly) > undeterministic behaviour. However, I agree it is easier to ensure for > something like UEFI than for a full-fledged OS of course. > >> >>> >>> Diverging from both of your concepts entirely and keeping it very >>> simple, what keeps us from simply ASSERTing based on return value? We >>> could introduce a CONSTRAINT_ASSERT or similar, but it would be >>> completely different from Vitaly's proposal. It would be a caller macro >>> to ASSERT based on a pre-defined list of return statuses. >>> To achieve this, we would order all statuses by types (classes). At the >>> moment, I can only think of two: "environmental" and "constraint". For >>> example, "Out Of Resources" would be environmental and "Invalid >>> Parameter" would be constraint. We could define environment statuses >>> explicitly (as it is less likely new environment statuses are >>> introduced) and define constraint statuses as "not environmental" (to >>> silently support new constraint statuses, however of course it is a >>> little error-prone with forwards-compatibility). As a bonus, it forces >>> developers to use truly appropiate error codes and correctly propagate >>> them from callees for this to work. :) >>> This solution would be backwards-compatible in a compilation sense as it >>> can simply be a new macro, however for its possible complexity (switch >>> cases) I might actually prefer a function. But it would not be >>> backwards-compatible with the current functionality, which none of the >>> "caller-based" concepts can be anyway (the caller needs explicit code). >>> >> >> It is an interesting idea to add granularity, but at the end of the >> knowledge of the correct behavior of the lower level library code really >> lives in code that calls this. I will admit I can see some value to >> making RETURN_INVALID_PARAMETER different than RETURN_BUFFER_TOO_SMALL. > > So far, I prefer this idea as the only requirement is that function > contracts are well-designed (i.e. reflect environmental vs constraint > errors by using sensical return values). > >> >> I kind of went down the C11 path (thanks for mentioning the event issue) >> but there are other ways to empower the caller. > > It's definitely always a good idea to stick to standards and > conventions. It appears my proposal is not conventional at all. However, > I also believe there sometimes is a more pragmatic approach to things > without imposing significant disadvantages, especially by limiting the > scope. > > To be honest, I'm now interested for which cases the C11 path is > advantagous over my approach, except that setting the constraint handler > is a one-time thing if it remains consistent throughout execution. I'd > imagine for some more complex handling logic I cannot think of an > example of (all we really want to do is ASSERT conditionally afterall), > and I hope this is not something edk2 would ever run into - keep it > simple in my opinion. > >> >> We could let the caller decide the behavior. That implies passing >> CHECK_DEFAULT (PCD), CHECK_ASSERT, or CHECK_NULL. >> >> Then keep backward compatibility. >> #define StrCpyS(Destination,DestMax,Source)StrCpuSC(Destination, >> DestMax, Source, CHECK_DEFAULT) >> >> But that seems like a lot of thrash to the BaseLib and a lot of new >> magic to teach developers. > > Fully agreed with the last sentence. That was one of the approaches I > mentioned before my proposal, but I included it only for completeness' > sake so we have a full overview. This requires prototype adaption for > literally every function that may be used in such a way (which scopes > beyond just BaseLib), ugly macros and more. Please don't do that. :) > >> >> I would guess that the most common usage of my library would be to turn >> Constraint Checking off in the entire driver or app and then handle the >> errors manually. > > I'm afraid that's what most platforms will end up with. Our main concern > was we may not allow such constraint violations to ASSERT, that is our > main objective. However, if it can be made advantageous to future coding > practice, it would be nice to have a decent solution for the caller to > have some freedom. More about that below. > >> >> On driver/app entry do: >> >> SetConstraintHandler (IgnoreConstraintHandler); >> >> Then handle the errors you care about. >> >> Status = StrCpyS (Destination,DestMax,Source); >> if (EFI_ERROR (Status)) { >> ASSERT_EFI_ERROR (Status); >> return Status; >> } >> >> or >> >> Status = StrCpyS (Destination,DestMax,Source); >> if (Status == RETURN_INVALID_PARAMETER) { >> ASSERT_EFI_ERROR (Status); >> return Status; >> } >> >> At the end of the day I've code my driver and I know the rules I coded >> to and I want predictable behavior. > > Yes, agreed. We want predictable behaviour of "not ASSERTing" for > untrusted input, because we *know* it is untrusted and the function > *can* (and should be able to) handle it - it is simply not a precondition. > >> >> I can see a Si vendor developing with ASSERT Constraint off and then >> when the customer turns it on stuff starts randomly breaking. > > A lot of Si and core (DxeCore, PeiCore, related libraries) code seems to > ASSERT (frequently *only* ASSERT with no actual handling, especially for > AllocatePool() == NULL) when there are valid error situations possible, > so I hope this approach would be a handy tool to satisfy their needs > without this absolutely terrible practice. :) > > > > I think I understood all your comment individually, but I'm afraid I'm > not sure what your suggested solution is right now. You commented > semi-positively on my proposal, you did not revoke your C11 path, and > you furthermore introduced CHECK_* - this is brainstorming, and I think > it's a great thing, but I'm simply not sure what your exact goal or > prefered route is right now. > > My proposal is a bit unconventional and I agree it sounds hacky, but so > far I'm not convinced it would be bad practice at all. It boils down to > sensefully using the information a function returns. This requires > nothing but the function to return sensical information, which every > function should anyway. > > Your proposal is close to the C standard, and that is a good thing first > of all. However, with the rest of edk2 not being very compliant, I don't > think it's an instant win as in "we just comply to standards" because > this choice does not suddenly make porting existing code significantly > easier compared to the total lack of standard library support for > anything but Shell apps. > Pragmatically, I think it solves the same problem equally well, but at a > higher cost (ensuring the implementation is safe and, strictly speaking, > an ever-so-slight runtime overhead). > >> >>> However, CONSTRAINT_ASSERT might actually be too specific. Maybe we want >>> something like "ASSERT_RETURN" and have the classes "ASSERT_CONSTRAINT" >>> and "ASSERT_ENVIRONMENT" (bitfield, like with DEBUG). The reason for >>> this is embedded systems where the environment is trusted/known and the >>> firmware is supposed to be well-matched. Think of a SoC with soldered >>> RAM - the firmware can very well make assumption about memory capacity >>> (considering usage across all modules in the worst case) and might >>> actually want ASSERTs on something like "Out Of Resources" because it's >>> supposed to be impossible within the bounds of this specific design. >>> It's possible this would need a new PCD's involvement, for example to >>> tell DxeCore whether to ASSERT on environmental aspects too. There could >>> be an argument-less macro that uses PCD config as base, and an >>> argument-based macro that uses only the parameters. Of course this >>> cannot cover everyone's very specific preferences as it's a per-module >>> switch, but it's the best I can think of right now. >>> >>> Something a bit unrelated now, but it would make this easier. You >>> mentioned PeCoffLib as example, and I think it's a good one, however it >>> has its own return status type within the context[1] which I will most >>> definitely be getting rid of as part of my upcoming (in the far-ish >>> future :) ) RFC. Could we expand RETURN_STATUS to have (very high?) >>> reserved values for caller-defined error codes to have all RETURN_STATUS >>> macros apply to those special return values too? We'd need to define >>> them categorically as "constraint status", but I don't see how a library >>> would declare a new environment status anyway. >>> >>> Regarding MdePkgCompatability.dsc.inc, I think one can override library >>> classes from the inc by declaring them after the !include statement, >>> please correct me if I'm wrong. If so, I strongly agree and think it >>> should be the case for all packages, so one only overrides the defaults >>> when something specific (debugging, performance-optimized versions, ...) >>> is required - easier to read, easier to maintain. The content is >>> needed/there anyway as the libraries are declared in the package's own >>> DSC. >>> >> >> Yes it the new .inc DSC file could be overridden by the platform DSC >> that includes it. >> >> I think this is a more general idea than something for this given patch. >> It is something we could make sure we update every stable tag or so, but >> I guess someone has to go 1st :). It would make it easier on platforms >> when they update the edk2 version. > > Agreed. > >> >> >> >> >>> It would be nice if you had comments regarding every aspect I just >>> mentioned, it was just something coming to my mind this morning. Thanks >>> for the input so far, it's nice to see some movement now! >>> >> >> Sorry for the delay > > Sure, thanks for taking time to respond! > >> >> Thanks, >> >> Andrew Fish >> >>> Best regards, >>> Marvin >>> >>> [1] >>> https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L20-L31 >>> https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L159 >>> >>> Am 16.02.2020 um 22:25 schrieb Andrew Fish via Groups.Io: >>>> Mike, >>>> >>>> Sorry I don't think I totally understood what felt wrong to me, so I did >>>> a bad job explaining my concerns. Also I don't think I was thinking >>>> enough in the context of the C11 StdLib. >>>> >>>> I think my concern is still the global scope of the constraint, even if >>>> we tune it per module. For example the DXE Core can interact with >>>> PE/COFF images and other data that could have indirectly come from the >>>> disk. So conceptually you may want to ASSERT on some constraints and not >>>> on others. I think that is why I ratholed on expanding the error >>>> handling macros as that was more in the vein of having the caller deal >>>> with it, so that is kind of like what you would do pre C11. Also the >>>> DebugLib is probably one of the MdePkg Libs that has the most instances >>>> floating around, so I think we should change it in a non backwards way >>>> very carefully. >>>> >>>> So after reading up on the C11 implementation of Constraints I think my >>>> alternate proposal is for us to add a ConstraintLib modeled after C11 >>>> vs. updating the DebugLib. This would solve the 2 things that made me >>>> uncomfortable: 1) Extending the DebugLib API; 2) Giving the caller >>>> control of the ASSERT behavior. It would still have the down side of >>>> breaking builds as the BaseLib would get a new dependency, so we could >>>> talk about adding these functions to the DebugLib as the cost of >>>> replicating code. >>>> >>>> C11 defines constraint_handler_t and set_constraint_handler_s as a way >>>> for the caller to configure the behavior for bounds checked functions. I >>>> think that is the behavior I prefer. So if we are going to make a change >>>> that impacts DebugLib compatibility I just want to make sure we have a >>>> conversation about all the options. My primary goal is we have the >>>> conversation, and if folks don't agree with me that is fine at least we >>>> talked about it. >>>> >>>> What I'm thinking about is as simply exposing an API to control the >>>> Constraint handler like C11. This could be done via an ConstrainLib or >>>> extending the DebugLib. >>>> >>>> The basic implementation of the lib would look like: >>>> >>>> typedef >>>> VOID >>>> (EFIAPI *CONSTRAINT_HANDLER) ( >>>> IN CONST CHAR8 *FileName, >>>> IN UINTN LineNumber, >>>> IN CONST CHAR8 *Description, >>>> IN EFI_STATUS Status >>>> ); >>>> >>>> >>>> // Default to AssertConstraintHandler to make it easier to implement >>>> Base and XIP libs. >>>> // We could have a PCD that also sets the default handler in a Lib >>>> Constructor. The default handler is implementation defined in C11. >>>> CONSTRAINT_HANDLER gDefaultConstraintHandler = AssertConstraintHandler; >>>> CONSTRAINT_HANDLER gActiveConstraintHandler = gDefaultConstraintHandler; >>>> >>>> BOOLEAN >>>> EFIAPI >>>> ConstraintAssertEnabled ( >>>> VOID >>>> ) >>>> { >>>> return (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & >>>> DEBUG_PROPERTY_DEBUG_CONSTRAINT_ENABLED) != 0); >>>> } >>>> >>>> EFI_STATUS >>>> EFIAPI >>>> SetConstraintHandler ( >>>> IN CONSTRAINT_HANDLER Handler >>>> ) >>>> { >>>> if (Handler == NULL) { >>>> gActiveConstraintHandler = gDefaultConstraintHandler; >>>> } else { >>>> gActiveConstraintHandler = Handler; >>>> } >>>> } >>>> >>>> VOID >>>> AssertConstraintHandler ( >>>> IN CONST CHAR8 *FileName, >>>> IN UINTN LineNumber, >>>> IN CONST CHAR8 *Description, >>>> IN EFI_STATUS Status >>>> ) >>>> { >>>> if (ConstraintAssertEnabled ()) { >>>> DEBUG ((EFI_D_ERROR, "\Constraint ASSERT (Status = %r): ", Status)); >>>> DebugAssert (FileName, LineNumber, Description) >>>> } >>>> >>>> return; >>>> } >>>> >>>> VOID >>>> IgnoreConstraintHandler ( >>>> IN CONST CHAR8 *FileName, >>>> IN UINTN LineNumber, >>>> IN CONST CHAR8 *Description, >>>> IN EFI_STATUS Status >>>> ) >>>> { >>>> return; >>>> } >>>> >>>> We could add macros for the code in the lib to call: >>>> >>>> #define CONSTRAINT_CHECK(Expression, Status) \ >>>> do { \ >>>> if (!(Expression)) { \ >>>> gActiveConstraintHandler (__FILE__, __LINE__, Expression, Status); \ >>>> return Status; \ >>>> } \ >>>> } while (FALSE) >>>> >>>> #define CONSTRAINT_REQUIRE(Expression, Status, Label) \ >>>> do { \ >>>> if (!(Expression)) { \ >>>> gActiveConstraintHandler (__FILE__, __LINE__, Expression, Status); \ >>>> goto Label; \ >>>> } \ >>>> } while (FALSE) >>>> >>>> >>>> As a caller we have now have control: >>>> EFI_STATUS Status; >>>> CHAR16 Dst[2]; >>>> >>>> SetConstraintHandler (IgnoreConstraintHandler); >>>> Status = StrCpyS (Dst, sizeof (Dst), L"Too Long"); >>>> Print (L"Dst =%s (%r)\n", Dst, Status); >>>> >>>> SetConstraintHandler (AssertConstraintHandler); >>>> StrCpyS (Dst, sizeof (Dst), L"Too Long"); >>>> >>>> Thanks, >>>> >>>> Andrew Fish >>>> >>>> PS Since I'm on a crazy idea roll another idea would be to add a >>>> MdePkgCompatability.dsc.inc file that could be used to future proof >>>> adding dependent libs to existing MdePkg libs. So a platform could >>>> include this .DSC and that would give them the default library mapping >>>> to keep code compiling. It will only work after other platforms start >>>> including it, but after that it would give default mappings for >>>> dependent libs. >>>> >>>> In our above example we could have added this and then existing builds >>>> that included MdePkgCompatability.dsc.inc would keep compiling. >>>> >>>> [LibraryClasses] >>>> >>>> DebugConstraintLib|MdePkg/Library/DebugConstraintLib/DebugConstraintLib.inf >>>> >>>> >>>> >>>>> On Feb 15, 2020, at 11:38 AM, Michael D Kinney >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com><mailto:michael.d.kinney@intel.com>> >>>>> wrote: >>>>> >>>>> Andrew, >>>>> I do not think of this as a globally scoped PCD.It can be set in >>>>> global scope in DSC.But is can also be set to values based on module >>>>> type or for specific modules.In the case of the safe string functions, >>>>> I think there is a desire to disable the constraint asserts when >>>>> building a UEFI App or UEFI Driver and implement those modules to >>>>> handle the error return values.Enabling the constraint asserts for PEI >>>>> Core, DXE Core, SMM/MM Core, PEIM, DXE, SMM/MM modules makes sense to >>>>> find incorrect input to these functions from modules that can >>>>> guarantee the inputs would never return an error and catch these as >>>>> part of dev/debug/validation builds. >>>>> I would not expect disabling on a module by module basis to be common. >>>>> I think the rule for API implementations is to only use >>>>> CONSTRAINT_ASSERT() for conditions that are also checked and return an >>>>> error or fail with predicable behavior that allows the system to >>>>> continue to function.ASSERT() is for conditions that the systemcan >>>>> notcontinue. >>>>> Best regards, >>>>> Mike >>>>> *From:*afish@apple.com >>>>> <mailto:afish@apple.com><mailto:afish@apple.com><afish@apple.com >>>>> <mailto:afish@apple.com> >>>>> <mailto:afish@apple.com>> >>>>> *Sent:*Friday, February 14, 2020 10:27 PM >>>>> *To:*vit9696 <vit9696@protonmail.com >>>>> <mailto:vit9696@protonmail.com><mailto:vit9696@protonmail.com>> >>>>> *Cc:*devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io><mailto:devel@edk2.groups.io>; Kinney, >>>>> Michael D <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>>; Gao, Liming >>>>> <liming.gao@intel.com >>>>> <mailto:liming.gao@intel.com><mailto:liming.gao@intel.com>>; Gao, >>>>> Zhichao >>>>> <zhichao.gao@intel.com >>>>> <mailto:zhichao.gao@intel.com><mailto:zhichao.gao@intel.com>>; >>>>> Marvin Häuser >>>>> <marvin.haeuser@outlook.com >>>>> <mailto:marvin.haeuser@outlook.com><mailto:marvin.haeuser@outlook.com>>; >>>>> Laszlo Ersek <lersek@redhat.com >>>>> <mailto:lersek@redhat.com><mailto:lersek@redhat.com>> >>>>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe >>>>> string constraint assertions >>>>> Vitaly, >>>>> Sorry after I sent the mail I realized it may come across as me >>>>> asking you to do work and that was not my intent. >>>>> I will point out though that a non backward compatible change to >>>>> something as fundamental as the DebugLib is a very big deal. I've got >>>>> a few different custom implementations that would break with this >>>>> change as Mike proposed. Given breaking every one's debug lib is such >>>>> a big deal maybe it is something that we should do as a long term plan >>>>> vs. some incremental fix. So my intent was to start a conversation >>>>> about what else we might want to change if we are going to break the >>>>> world. The only think worse than breaking the world is breaking the >>>>> world frequently. >>>>> I'm also a little worried that we are taking things that are today >>>>> locally scoped like SAFE_STRING_CONSTRAINT_CHECK and >>>>> SAFE_PRINT_CONSTRAINT_CHECK and making them global constructs. I think >>>>> the way others have dealt with things like this is to make them be >>>>> DEBUG prints vs. ASSERTs. Also even something as simple as >>>>> SAFE_STRING_CONSTRAINT_CHECK could be called from code that wants >>>>> ASSERT and CONSTRAINT_ASSERT behavior. It is not clear to me that the >>>>> low level code knows the right thing to do in a global sense even if >>>>> there is a PCD. It almost seems like we should have wrappers for the >>>>> Safe string functions that implement the behavior you want as a >>>>> caller. I'm not sure about that, but it seems like it is worth talking >>>>> about? >>>>> Thanks, >>>>> Andrew Fish >>>>> >>>>> >>>>> On Feb 14, 2020, at 7:31 PM, vit9696 <vit9696@protonmail.com >>>>> <mailto:vit9696@protonmail.com> >>>>> <mailto:vit9696@protonmail.com>> wrote: >>>>> Hi Andrew, >>>>> While your suggestions look interesting, I am afraid they are not >>>>> particularly what we want to cover with this discussion at the >>>>> moment. >>>>> Making assertions go through DEBUG printing functions/macros is >>>>> what we have to strongly disagree about. Assertions and debug >>>>> prints are separate things configurable by separate PCDs. We >>>>> should not mix them. Introducing constraint assertions is a >>>>> logical step forward in understanding that different software >>>>> works in different environments. >>>>> >>>>> * There are normal, or, as I call them, invariant >>>>> assertions (e.g. preconditions), for places where the function >>>>> cannot work unless the assertion is satisfied. This is where >>>>> we ASSERT. >>>>> * There are constraint assertions, which signalise that bad data >>>>> came through the function, even though the function was called >>>>> from a trusted source. This is where we call CONSTRAINT_ASSERT. >>>>> >>>>> The thing we need is to have the latter separable >>>>> and configurable, because not every piece of software works in a >>>>> trusted environment. Other than that, constraint assertions, when >>>>> enabled, are not anyhow different from normal assertions in the >>>>> sense of action taken. Assertions have configurable breakpoints >>>>> and deadloops, and DEBUG prints go through a different route in >>>>> DebugLib that may cause entirely different effects. For example, >>>>> we halt execution upon printing to DEBUG_ERROR in our DebugLib >>>>> even in release builds. >>>>> >>>>> =To make it clear, currently I plan to add the following interface: >>>>> #define CONSTRAINT_ASSERT(Expression) \ >>>>> do { \ >>>>> if (DebugConstraintAssertEnabled ()) { \ >>>>> if (!(Expression)) { \ >>>>> _ASSERT (Expression); \ >>>>> ANALYZER_UNREACHABLE (); \ >>>>> } \ >>>>> } \ >>>>> } while (FALSE) >>>>> with DebugConstraintAssertEnabled implemented as >>>>> (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & >>>>> DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED | >>>>> DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED) == >>>>> DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED) >>>>> Your suggestion with require macros looks interesting indeed, but >>>>> I believe it is in fact parallel to this discussion. The change we >>>>> discuss introduces a new assertion primitive — constraint >>>>> assertions, while REQUIRE macros are mostly about advanced syntax >>>>> sugar and higher level assertion primitives on top of existing >>>>> ones. Perhaps we can have this and make a good use of it, >>>>> especially given that it brought some practical benefit in Apple, >>>>> but I would rather discuss this later once constraint assertions >>>>> are merged into EDK II tree. >>>>> Best wishes, >>>>> Vitaly >>>>> On Sat, Feb 15, 2020 at 03:02, Andrew Fish <afish@apple.com >>>>> <mailto:afish@apple.com> >>>>> <mailto:afish@apple.com>> wrote: >>>>> >>>>> >>>>> >>>>> On Feb 14, 2020, at 2:50 PM, Michael D Kinney >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> wrote: >>>>> Hi Vitaly, >>>>> I agree that this proposal makes a lot of sense. We >>>>> recently added a new assert type called STATIC_ASSERT() >>>>> for assert conditions that can be tested at build time. >>>>> A new assert type for checks that can be removed and the >>>>> API still guarantees that it fails gracefully with a >>>>> proper return code is a good idea. Given we have >>>>> STATIC_ASSERT(), how about naming the new macro >>>>> CONSTRAINT_ASSERT()? >>>>> We also want the default to be enabled. The current use of >>>>> bit 0x40 inPcdDebugPropertyMask is always clear, so we >>>>> want the asserts enabled when 0x40 is clear. We can change >>>>> the name of the define bit to >>>>> DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED so bit 0x40 needs >>>>> to be set inPcdDebugPropertyMaskto disable these types of >>>>> asserts. >>>>> This approach does require all theDebugLibimplementations >>>>> to be updated with the newDebugConstraintAssertDisabled() >>>>> API. >>>>> >>>>> Mike, >>>>> If you wanted to be backward compatible you could just >>>>> use DebugAssertEnabled () but in place of _ASSERT() use >>>>> _CONSTRAINT_ASSERT >>>>> #define _ASSERT(Expression) DebugAssert (__FILE__, __LINE__, >>>>> #Expression) >>>>> #define _CONSTRAINT_ASSERT(Expression) DebugPrint >>>>> (DEBUG_ERROR, "ASSERT %a(%d): %a\n",, __FILE__, __LINE__, >>>>> #Expression) >>>>> Not as elegant as the non backward compatible change, but I >>>>> thought I'd throw it out there. >>>>> There are some ancient Apple C ASSERT macros [AssertMacros.h] >>>>> that also have the concept of require. Require includes an >>>>> exception label (goto label). It is like a CONSTRAINT_ASSERT() >>>>> but with the label. On release builds the DEBUG prints are >>>>> skipped. >>>>> So we could do something like: >>>>> EFI_STATUS Status = EFI_INVALID_PARAMETER; >>>>> REQUIRE(Arg1 != NULL, ErrorExit); >>>>> REQUIRE(Arg2 != NULL, ErrorExit); >>>>> REQUIRE(Arg3 != NULL, ErrorExit); >>>>> ErrorExit: >>>>> return Status; >>>>> There is another form that allows an ACTION (a statement to >>>>> execute. So you could have: >>>>> EFI_STATUS Status; >>>>> REQUIRE_ACTION(Arg1 != NULL, ErrorExit, Status = >>>>> EFI_INVALID_PARAMETER); >>>>> REQUIRE_ACTION(Arg2 != NULL, ErrorExit, Status = >>>>> EFI_INVALID_PARAMETER); >>>>> REQUIRE_ACTION(Arg3 != NULL, ErrorExit, Status = >>>>> EFI_INVALID_PARAMETER); >>>>> ErrorExit: >>>>> return Status; >>>>> If you use CONSTRAINT_ASSERT(); >>>>> if (Arg1 == NULL || Arg2 == NULL || Arg3 == NULL) { >>>>> CONSTRAINT_ASSERT (Arg1 != NULL); >>>>> CONSTRAINT_ASSERT (Arg2 != NULL); >>>>> CONSTRAINT_ASSERT (Arg3 != NULL); >>>>> return EFI_INVALID_PARAMETER; >>>>> } >>>>> I'd note error processing args on entry is the simplest case. >>>>> In a more complex case when cleanup is required the goto >>>>> label is more useful. >>>>> I guess we could argue for less typing and more symmetry and >>>>> say use ASSERT, CONSTRAINT, and REQUIRE. I guess you could add >>>>> an ASSERT_ACTION too. >>>>> The AssertMacros.h versions also support _quiet (skip the >>>>> print) and _string (add a string to the print) so you end up >>>>> with: >>>>> REQUIRE >>>>> REQUIRE_STRING >>>>> REQUIRE_QUIET >>>>> REQUIRE_ACTION >>>>> REQUIRE_ACTION_STRING >>>>> REQUIRE_ACTION_QUIET >>>>> We could also end up with >>>>> CONSTRAINT >>>>> CONSTRAINT_STRING >>>>> CONSTRAINT_QUIET >>>>> I think the main idea behind _QUIET is you can silence things >>>>> that are too noisy, and you can easily make noise things show >>>>> up by removing the _QUIET to debug. >>>>> I'd thought I throw out the other forms for folks to think >>>>> about. I'm probably biased as I used to looking at code and >>>>> seeing things like require_action_string(Arg1 != NULL, >>>>> ErrorExit, Status = EFI_INVALID_PARAMETER, "1st Arg1 check"); >>>>> Thanks, >>>>> Andrew Fish >>>>> PS The old debug macros had 2 versions of CONSTRAINT check and >>>>> verify. The check version was compiled out on a release build, >>>>> the verify version always does the check and just skips the >>>>> DEBUG print. >>>>> >>>>> Mike >>>>> *From:*vit9696 <vit9696@protonmail.com >>>>> <mailto:vit9696@protonmail.com> >>>>> <mailto:vit9696@protonmail.com>> >>>>> *Sent:*Friday, February 14, 2020 9:38 AM >>>>> *To:*Kinney, Michael D <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> >>>>> *Cc:*devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io><mailto:devel@edk2.groups.io>; >>>>> Gao, Liming <liming.gao@intel.com >>>>> <mailto:liming.gao@intel.com> >>>>> <mailto:liming.gao@intel.com>>; Gao, Zhichao >>>>> <zhichao.gao@intel.com >>>>> <mailto:zhichao.gao@intel.com><mailto:zhichao.gao@intel.com>>; >>>>> Marvin Häuser <marvin.haeuser@outlook.com >>>>> <mailto:marvin.haeuser@outlook.com> >>>>> <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek >>>>> <lersek@redhat.com >>>>> <mailto:lersek@redhat.com><mailto:lersek@redhat.com>> >>>>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>>> disable safe string constraint assertions >>>>> Michael, >>>>> Generalising the approach makes good sense to me, but we >>>>> need to make an obvious distinguishable difference between: >>>>> - precondition and invariant assertions (i.e. assertions, >>>>> where function will NOT work if they are violated) >>>>> - constraint asserts (i.e. assertions, which allow us to >>>>> spot unintentional behaviour when parsing untrusted data, >>>>> but which do not break function behaviour). >>>>> As we want to use this outside of SafeString, I suggest >>>>> the following: >>>>> - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED 0x40 >>>>> bit for PcdDebugPropertyMask instead >>>>> of PcdAssertOnSafeStringConstraints. >>>>> - Introduce DebugAssertConstraintEnabled DebugLib function >>>>> to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. >>>>> - Introduce ASSERT_CONSTRAINT macro, that will assert only >>>>> if DebugConstraintAssertEnabled returns true. >>>>> - Change SafeString ASSERTS >>>>> in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. >>>>> - Use ASSERT_CONSTRAINT in other places where necessary. >>>>> >>>>> I believe this way lines best with EDK II design. If there >>>>> are no objections, I can submit the patch in the beginning >>>>> of next week. >>>>> >>>>> Best wishes, >>>>> Vitaly >>>>> >>>>> 14 февр. 2020 г., в 20:00, Kinney, Michael D >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> написал(а): >>>>> Vitaly, >>>>> I want to make sure a feature PCD can be used to >>>>> disable ASSERT() behavior in more than just safe >>>>> string functions inBaseLib. >>>>> Can we consider changing the name and description >>>>> ofPcdAssertOnSafeStringConstraintsto be more generic, >>>>> so if we find other lib APIs, the name will make sense? >>>>> Maybe something like:PcdEnableLibraryAssertChecks? >>>>> Default is TRUE. Can set to FALSE in DSC file to >>>>> disable ASSERT() checks. >>>>> Thanks, >>>>> Mike >>>>> *From:*devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io><devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io>>*On Behalf Of*Vitaly >>>>> Cheptsov via Groups.Io >>>>> *Sent:*Friday, February 14, 2020 3:55 AM >>>>> *To:*Kinney, Michael D <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>>; Gao, Liming >>>>> <liming.gao@intel.com >>>>> <mailto:liming.gao@intel.com><mailto:liming.gao@intel.com>>; >>>>> Gao, Zhichao <zhichao.gao@intel.com >>>>> <mailto:zhichao.gao@intel.com> >>>>> <mailto:zhichao.gao@intel.com>>;devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io> >>>>> *Cc:*Marvin Häuser <marvin.haeuser@outlook.com >>>>> <mailto:marvin.haeuser@outlook.com> >>>>> <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek >>>>> <lersek@redhat.com >>>>> <mailto:lersek@redhat.com><mailto:lersek@redhat.com>> >>>>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>>> disable safe string constraint assertions >>>>> Replying as per Liming's request for this to be merged >>>>> into edk2-stable202002. >>>>> On Mon, Feb 10, 2020 at 14:12, vit9696 >>>>> <vit9696@protonmail.com <mailto:vit9696@protonmail.com> >>>>> <mailto:vit9696@protonmail.com>> wrote: >>>>> >>>>> Hello, >>>>> >>>>> It has been quite some time since we submitted the >>>>> patch with so far no negative response. As I >>>>> mentioned previously, my team will strongly >>>>> benefit from its landing in EDK II mainline. Since >>>>> it does not add any regressions and can be viewed >>>>> as a feature implementation for the rest of EDK II >>>>> users, I request this to be merged upstream in >>>>> edk2-stable202002. >>>>> >>>>> Best wishes, >>>>> Vitaly >>>>> >>>>>> 27 янв. 2020 г., в 12:47, vit9696 >>>>> <vit9696@protonmail.com >>>>> <mailto:vit9696@protonmail.com> >>>>> <mailto:vit9696@protonmail.com>> написал(а): >>>>>> >>>>>> >>>>>> Hi Mike, >>>>>> >>>>>> Any progress with this? We would really benefit >>>>> from this landing in the next stable release. >>>>>> >>>>>> Best, >>>>>> Vitaly >>>>>> >>>>>>> 8 янв. 2020 г., в 19:35, Kinney, Michael D >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> написал(а): >>>>>>> >>>>>>> >>>>>>> Hi Vitaly, >>>>>>> >>>>>>> Thanks for the additional background. I would like >>>>>>> a couple extra day to review the PCD name and >>>>> the places >>>>>>> the PCD might potentially be used. >>>>>>> >>>>>>> If we find other APIs where ASSERT() behavior >>>>> is only >>>>>>> valuable during dev/debug to quickly identify >>>>> misuse >>>>>>> with trusted data and the API provides predicable >>>>>>> return behavior when ASSERT() is disabled, then >>>>> I would >>>>>>> like to have a pattern we can potentially apply >>>>> to all >>>>>>> these APIs across all packages. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Mike >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io><devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io>> On >>>>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>>>> Sent: Monday, January 6, 2020 10:44 AM >>>>>>>> To: Kinney, Michael D >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> >>>>>>>> Cc:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io> >>>>>>>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add >>>>> PCD to >>>>>>>> disable safe string constraint assertions >>>>>>>> >>>>>>>> Hi Mike, >>>>>>>> >>>>>>>> Yes, the primary use case is for UEFI >>>>> Applications. We >>>>>>>> do not want to disable ASSERT’s completely, as >>>>>>>> assertions that make sense, i.e. the ones >>>>> signalising >>>>>>>> about interface misuse, are helpful for debugging. >>>>>>>> >>>>>>>> I have already explained in the BZ that >>>>> basically all >>>>>>>> safe string constraint assertions make no >>>>> sense for >>>>>>>> handling untrusted data. We find this use case >>>>> very >>>>>>>> logical, as these functions behave properly with >>>>>>>> assertions disabled and cover all these error >>>>>>>> conditions by the return statuses. In such >>>>> situation is >>>>>>>> not useful for these functions to assert, as >>>>> we end up >>>>>>>> inefficiently reimplementing the logic. I >>>>> would have >>>>>>>> liked the approach of discussing the interfaces >>>>>>>> individually, but I struggle to find any that >>>>> makes >>>>>>>> sense from this point of view. >>>>>>>> >>>>>>>> AsciiStrToGuid will ASSERT when the length of the >>>>>>>> passed string is odd. Functions that cannot, ahem, >>>>>>>> parse, for us are pretty much useless. >>>>>>>> AsciiStrCatS will ASSERT when the appended >>>>> string does >>>>>>>> not fit the buffer. For us this logic makes this >>>>>>>> function pretty much equivalent to deprecated >>>>> and thus >>>>>>>> unavailable AsciiStrCat, except it is also slower. >>>>>>>> >>>>>>>> My original suggestion was to remove the >>>>> assertions >>>>>>>> entirely, but several people here said that >>>>> they use >>>>>>>> them to verify usage errors when handling >>>>> trusted data. >>>>>>>> This makes good sense to me, so we suggest to >>>>> support >>>>>>>> both cases by introducing a PCD in this patch. >>>>>>>> >>>>>>>> Best wishes, >>>>>>>> Vitaly >>>>>>>> >>>>>>>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >>>>>>>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> написал(а): >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi Vitaly, >>>>>>>>> >>>>>>>>> Is the use case for UEFI Applications? >>>>>>>>> >>>>>>>>> There is a different mechanism to disable all >>>>>>>> ASSERT() >>>>>>>>> statements within a UEFI Application. >>>>>>>>> >>>>>>>>> If a component is consuming data from an >>>>> untrusted >>>>>>>> source, >>>>>>>>> then that component is required to verify the >>>>>>>> untrusted >>>>>>>>> data before passing it to a function that clearly >>>>>>>> documents >>>>>>>>> is input requirements. If this approach is >>>>> followed, >>>>>>>> then >>>>>>>>> the BaseLib functions can be used "as is" as >>>>> long as >>>>>>>> the >>>>>>>>> ASSERT() conditions are verified before calling. >>>>>>>>> >>>>>>>>> If there are some APIs that currently >>>>> document their >>>>>>>> ASSERT() >>>>>>>>> behavior and we think that ASSERT() behavior is >>>>>>>> incorrect and >>>>>>>>> should be handled by an existing error return >>>>> value, >>>>>>>> then we >>>>>>>>> should discuss each of those APIs individually. >>>>>>>>> >>>>>>>>> Mike >>>>>>>>> >>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io><devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io>> On >>>>>>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>>>>>> Sent: Friday, January 3, 2020 9:13 AM >>>>>>>>>> To:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io> >>>>>>>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>>>>>> disable >>>>>>>>>> safe string constraint assertions >>>>>>>>>> >>>>>>>>>> REF: >>>>>>>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >>>>>>>>>> >>>>>>>>>> Requesting for merge in edk2-stable202002. >>>>>>>>>> >>>>>>>>>> Changes since V1: >>>>>>>>>> - Enable assertions by default to preserve the >>>>>>>> original >>>>>>>>>> behaviour >>>>>>>>>> - Fix bugzilla reference link >>>>>>>>>> - Update documentation in BaseLib.h >>>>>>>>>> >>>>>>>>>> Vitaly Cheptsov (1): >>>>>>>>>> MdePkg: Add PCD to disable safe string >>>>> constraint >>>>>>>>>> assertions >>>>>>>>>> >>>>>>>>>> MdePkg/MdePkg.dec | 6 ++ >>>>>>>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >>>>>>>>>> MdePkg/Include/Library/BaseLib.h | 74 >>>>>>>>>> +++++++++++++------- >>>>>>>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >>>>>>>>>> MdePkg/MdePkg.uni | 6 ++ >>>>>>>>>> 5 files changed, 71 insertions(+), 30 >>>>> deletions(-) >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> 2.21.0 (Apple Git-122.2) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -=-=-=-=-=-= >>>>>>>>>> Groups.io <http://groups.io/><http://groups.io/>Links: You >>>>> receive all messages sent to >>>>>>>> this >>>>>>>>>> group. >>>>>>>>>> >>>>>>>>>> View/Reply Online (#52837): >>>>>>>>>> https://edk2.groups.io/g/devel/message/52837 >>>>>>>>>> Mute This Topic: >>>>>>>> https://groups.io/mt/69401948/1643496 >>>>>>>>>> Group Owner:devel+owner@edk2.groups.io >>>>>>>>>> <mailto:devel+owner@edk2.groups.io> >>>>> <mailto:devel+owner@edk2.groups.io> >>>>>>>>>> Unsubscribe:https://edk2.groups.io/g/devel/unsub >>>>>>>>>> [michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>] >>>>>>>>>> -=-=-=-=-=-= >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>> >>> >> ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-03-03 19:38 ` Marvin Häuser @ 2020-03-04 0:19 ` Liming Gao 0 siblings, 0 replies; 28+ messages in thread From: Liming Gao @ 2020-03-04 0:19 UTC (permalink / raw) To: devel@edk2.groups.io, mhaeuser@outlook.de, Andrew Fish Cc: Kinney, Michael D, vit9696, Gao, Zhichao, Laszlo Ersek Marvin: I will add this feature into next stable tag planning. Let's try to finish it for next stable tag. Thanks Liming -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin Häuser Sent: 2020年3月4日 3:39 To: Andrew Fish <afish@apple.com>; devel@edk2.groups.io Cc: Kinney, Michael D <michael.d.kinney@intel.com>; vit9696 <vit9696@protonmail.com>; Gao, Liming <liming.gao@intel.com>; Gao, Zhichao <zhichao.gao@intel.com>; Laszlo Ersek <lersek@redhat.com> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions Good day, Unfortunately, this discussion has died again. This is fine by me, I'm also fine with the already proposed patch and mostly shared different approaches based on the vastly diverging feedback, but it works for us either way. Can the patch please be reviewed and accepted for the next stable tag? Thanks! Best regards, Marvin Am 20.02.2020 um 11:18 schrieb Marvin Häuser: > Hey Andrew, > > Thanks once again for your comments, mine are inline. > > Best regards, > Marvin > > Am 20.02.2020 um 00:55 schrieb Andrew Fish: >> >> >>> On Feb 17, 2020, at 12:26 AM, Marvin Häuser <mhaeuser@outlook.de >>> <mailto:mhaeuser@outlook.de>> wrote: >>> >>> Good day Andrew, >>> >>> First of all, thank you very much for putting this amount of thought >>> into the situation. I definitely agree with the problem you see, but >>> I could also live with Vitaly's proposal. However, I think you are >>> overcomplicating the situation a little. So, we do agree the caller >>> needs control of the behaviour, however your proposal needs "support" >>> from both the caller (choose the handler) and the callee (call the >>> handler) and is neither thread-safe nor event-safe as Vitaly already >>> pointed out. Fixing these problems would only worsen the complexity >>> situation in my opinion. >> >> Well EFI does not have Threads but it does have Events. I agree if an > > Sorry, we were refering to the UEFI Multi-Processor services, so > threads in a physical sense. I think very similar concerns apply > regarding these services as with software multithreading in this > context, please correct me if I'm wrong. > >> Event handler changes the constraint handler it would need to restore >> it so my proposal is missing an API. >> >> CONSTRAINT_HANDLER * >> GetConstraintHandler ( >> VOID >> ) >> { >> return gActiveConstraintHandler; >> } >> >> You an always use the standard EFI >> >> It is important to remember that all the EFI images >> (drivers/applications) are statically linked and they carry a unique >> instance of the Debug (or new Constraint) lib. So in your driver or >> App is very self contained. Also a lot of the events are phase >> related callbacks, and since there are no threads your drivers main >> thread is only really running when your driver, or app, dispatches. A >> lot of the callbacks are via protocols your driver publishes, but >> they have strict TPL rules so you can prevent reentrancy in general. >> So there is a lot more determinism in EFI vs. generic threaded coding. > > This is true, but yet all edge-cases should be covered (event > "interruption", actual (processor) interrupts, Multi-Processor > execution), so developers do not run into unexpected (seemingly) > undeterministic behaviour. However, I agree it is easier to ensure for > something like UEFI than for a full-fledged OS of course. > >> >>> >>> Diverging from both of your concepts entirely and keeping it very >>> simple, what keeps us from simply ASSERTing based on return value? >>> We could introduce a CONSTRAINT_ASSERT or similar, but it would be >>> completely different from Vitaly's proposal. It would be a caller >>> macro to ASSERT based on a pre-defined list of return statuses. >>> To achieve this, we would order all statuses by types (classes). At >>> the moment, I can only think of two: "environmental" and >>> "constraint". For example, "Out Of Resources" would be environmental >>> and "Invalid Parameter" would be constraint. We could define >>> environment statuses explicitly (as it is less likely new >>> environment statuses are >>> introduced) and define constraint statuses as "not environmental" >>> (to silently support new constraint statuses, however of course it >>> is a little error-prone with forwards-compatibility). As a bonus, it >>> forces developers to use truly appropiate error codes and correctly >>> propagate them from callees for this to work. :) This solution would >>> be backwards-compatible in a compilation sense as it can simply be a >>> new macro, however for its possible complexity (switch >>> cases) I might actually prefer a function. But it would not be >>> backwards-compatible with the current functionality, which none of >>> the "caller-based" concepts can be anyway (the caller needs explicit code). >>> >> >> It is an interesting idea to add granularity, but at the end of the >> knowledge of the correct behavior of the lower level library code >> really lives in code that calls this. I will admit I can see some >> value to making RETURN_INVALID_PARAMETER different than RETURN_BUFFER_TOO_SMALL. > > So far, I prefer this idea as the only requirement is that function > contracts are well-designed (i.e. reflect environmental vs constraint > errors by using sensical return values). > >> >> I kind of went down the C11 path (thanks for mentioning the event >> issue) but there are other ways to empower the caller. > > It's definitely always a good idea to stick to standards and > conventions. It appears my proposal is not conventional at all. > However, I also believe there sometimes is a more pragmatic approach > to things without imposing significant disadvantages, especially by > limiting the scope. > > To be honest, I'm now interested for which cases the C11 path is > advantagous over my approach, except that setting the constraint > handler is a one-time thing if it remains consistent throughout > execution. I'd imagine for some more complex handling logic I cannot > think of an example of (all we really want to do is ASSERT > conditionally afterall), and I hope this is not something edk2 would > ever run into - keep it simple in my opinion. > >> >> We could let the caller decide the behavior. That implies passing >> CHECK_DEFAULT (PCD), CHECK_ASSERT, or CHECK_NULL. >> >> Then keep backward compatibility. >> #define StrCpyS(Destination,DestMax,Source)StrCpuSC(Destination, >> DestMax, Source, CHECK_DEFAULT) >> >> But that seems like a lot of thrash to the BaseLib and a lot of new >> magic to teach developers. > > Fully agreed with the last sentence. That was one of the approaches I > mentioned before my proposal, but I included it only for completeness' > sake so we have a full overview. This requires prototype adaption for > literally every function that may be used in such a way (which scopes > beyond just BaseLib), ugly macros and more. Please don't do that. :) > >> >> I would guess that the most common usage of my library would be to >> turn Constraint Checking off in the entire driver or app and then >> handle the errors manually. > > I'm afraid that's what most platforms will end up with. Our main > concern was we may not allow such constraint violations to ASSERT, > that is our main objective. However, if it can be made advantageous to > future coding practice, it would be nice to have a decent solution for > the caller to have some freedom. More about that below. > >> >> On driver/app entry do: >> >> SetConstraintHandler (IgnoreConstraintHandler); >> >> Then handle the errors you care about. >> >> Status = StrCpyS (Destination,DestMax,Source); if (EFI_ERROR >> (Status)) { ASSERT_EFI_ERROR (Status); return Status; } >> >> or >> >> Status = StrCpyS (Destination,DestMax,Source); if (Status == >> RETURN_INVALID_PARAMETER) { ASSERT_EFI_ERROR (Status); return Status; >> } >> >> At the end of the day I've code my driver and I know the rules I >> coded to and I want predictable behavior. > > Yes, agreed. We want predictable behaviour of "not ASSERTing" for > untrusted input, because we *know* it is untrusted and the function > *can* (and should be able to) handle it - it is simply not a precondition. > >> >> I can see a Si vendor developing with ASSERT Constraint off and then >> when the customer turns it on stuff starts randomly breaking. > > A lot of Si and core (DxeCore, PeiCore, related libraries) code seems > to ASSERT (frequently *only* ASSERT with no actual handling, > especially for > AllocatePool() == NULL) when there are valid error situations > possible, so I hope this approach would be a handy tool to satisfy > their needs without this absolutely terrible practice. :) > > > > I think I understood all your comment individually, but I'm afraid I'm > not sure what your suggested solution is right now. You commented > semi-positively on my proposal, you did not revoke your C11 path, and > you furthermore introduced CHECK_* - this is brainstorming, and I > think it's a great thing, but I'm simply not sure what your exact goal > or prefered route is right now. > > My proposal is a bit unconventional and I agree it sounds hacky, but > so far I'm not convinced it would be bad practice at all. It boils > down to sensefully using the information a function returns. This > requires nothing but the function to return sensical information, > which every function should anyway. > > Your proposal is close to the C standard, and that is a good thing > first of all. However, with the rest of edk2 not being very compliant, > I don't think it's an instant win as in "we just comply to standards" > because this choice does not suddenly make porting existing code > significantly easier compared to the total lack of standard library > support for anything but Shell apps. > Pragmatically, I think it solves the same problem equally well, but at > a higher cost (ensuring the implementation is safe and, strictly > speaking, an ever-so-slight runtime overhead). > >> >>> However, CONSTRAINT_ASSERT might actually be too specific. Maybe we >>> want something like "ASSERT_RETURN" and have the classes "ASSERT_CONSTRAINT" >>> and "ASSERT_ENVIRONMENT" (bitfield, like with DEBUG). The reason for >>> this is embedded systems where the environment is trusted/known and >>> the firmware is supposed to be well-matched. Think of a SoC with >>> soldered RAM - the firmware can very well make assumption about >>> memory capacity (considering usage across all modules in the worst >>> case) and might actually want ASSERTs on something like "Out Of >>> Resources" because it's supposed to be impossible within the bounds of this specific design. >>> It's possible this would need a new PCD's involvement, for example >>> to tell DxeCore whether to ASSERT on environmental aspects too. >>> There could be an argument-less macro that uses PCD config as base, >>> and an argument-based macro that uses only the parameters. Of course >>> this cannot cover everyone's very specific preferences as it's a >>> per-module switch, but it's the best I can think of right now. >>> >>> Something a bit unrelated now, but it would make this easier. You >>> mentioned PeCoffLib as example, and I think it's a good one, however >>> it has its own return status type within the context[1] which I will >>> most definitely be getting rid of as part of my upcoming (in the >>> far-ish future :) ) RFC. Could we expand RETURN_STATUS to have (very >>> high?) reserved values for caller-defined error codes to have all >>> RETURN_STATUS macros apply to those special return values too? We'd >>> need to define them categorically as "constraint status", but I >>> don't see how a library would declare a new environment status anyway. >>> >>> Regarding MdePkgCompatability.dsc.inc, I think one can override >>> library classes from the inc by declaring them after the !include >>> statement, please correct me if I'm wrong. If so, I strongly agree >>> and think it should be the case for all packages, so one only >>> overrides the defaults when something specific (debugging, >>> performance-optimized versions, ...) is required - easier to read, >>> easier to maintain. The content is needed/there anyway as the >>> libraries are declared in the package's own DSC. >>> >> >> Yes it the new .inc DSC file could be overridden by the platform DSC >> that includes it. >> >> I think this is a more general idea than something for this given patch. >> It is something we could make sure we update every stable tag or so, >> but I guess someone has to go 1st :). It would make it easier on >> platforms when they update the edk2 version. > > Agreed. > >> >> >> >> >>> It would be nice if you had comments regarding every aspect I just >>> mentioned, it was just something coming to my mind this morning. >>> Thanks for the input so far, it's nice to see some movement now! >>> >> >> Sorry for the delay > > Sure, thanks for taking time to respond! > >> >> Thanks, >> >> Andrew Fish >> >>> Best regards, >>> Marvin >>> >>> [1] >>> https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39 >>> bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L20-L31 >>> https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39 >>> bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L159 >>> >>> Am 16.02.2020 um 22:25 schrieb Andrew Fish via Groups.Io: >>>> Mike, >>>> >>>> Sorry I don't think I totally understood what felt wrong to me, so >>>> I did a bad job explaining my concerns. Also I don't think I was >>>> thinking enough in the context of the C11 StdLib. >>>> >>>> I think my concern is still the global scope of the constraint, >>>> even if we tune it per module. For example the DXE Core can >>>> interact with PE/COFF images and other data that could have >>>> indirectly come from the disk. So conceptually you may want to >>>> ASSERT on some constraints and not on others. I think that is why I >>>> ratholed on expanding the error handling macros as that was more in >>>> the vein of having the caller deal with it, so that is kind of like >>>> what you would do pre C11. Also the DebugLib is probably one of the >>>> MdePkg Libs that has the most instances floating around, so I think >>>> we should change it in a non backwards way very carefully. >>>> >>>> So after reading up on the C11 implementation of Constraints I >>>> think my alternate proposal is for us to add a ConstraintLib >>>> modeled after C11 vs. updating the DebugLib. This would solve the 2 >>>> things that made me >>>> uncomfortable: 1) Extending the DebugLib API; 2) Giving the caller >>>> control of the ASSERT behavior. It would still have the down side >>>> of breaking builds as the BaseLib would get a new dependency, so we >>>> could talk about adding these functions to the DebugLib as the cost >>>> of replicating code. >>>> >>>> C11 defines constraint_handler_t and set_constraint_handler_s as a >>>> way for the caller to configure the behavior for bounds checked >>>> functions. I think that is the behavior I prefer. So if we are >>>> going to make a change that impacts DebugLib compatibility I just >>>> want to make sure we have a conversation about all the options. My >>>> primary goal is we have the conversation, and if folks don't agree >>>> with me that is fine at least we talked about it. >>>> >>>> What I'm thinking about is as simply exposing an API to control the >>>> Constraint handler like C11. This could be done via an ConstrainLib >>>> or extending the DebugLib. >>>> >>>> The basic implementation of the lib would look like: >>>> >>>> typedef >>>> VOID >>>> (EFIAPI *CONSTRAINT_HANDLER) ( >>>> IN CONST CHAR8 *FileName, >>>> IN UINTN LineNumber, IN CONST CHAR8 *Description, >>>> IN EFI_STATUS Status ); >>>> >>>> >>>> // Default to AssertConstraintHandler to make it easier to >>>> implement Base and XIP libs. >>>> // We could have a PCD that also sets the default handler in a Lib >>>> Constructor. The default handler is implementation defined in C11. >>>> CONSTRAINT_HANDLER gDefaultConstraintHandler = >>>> AssertConstraintHandler; CONSTRAINT_HANDLER >>>> gActiveConstraintHandler = gDefaultConstraintHandler; >>>> >>>> BOOLEAN >>>> EFIAPI >>>> ConstraintAssertEnabled ( >>>> VOID >>>> ) >>>> { >>>> return (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & >>>> DEBUG_PROPERTY_DEBUG_CONSTRAINT_ENABLED) != 0); } >>>> >>>> EFI_STATUS >>>> EFIAPI >>>> SetConstraintHandler ( >>>> IN CONSTRAINT_HANDLER Handler >>>> ) >>>> { >>>> if (Handler == NULL) { >>>> gActiveConstraintHandler = gDefaultConstraintHandler; } else { >>>> gActiveConstraintHandler = Handler; } } >>>> >>>> VOID >>>> AssertConstraintHandler ( >>>> IN CONST CHAR8 *FileName, >>>> IN UINTN LineNumber, IN CONST CHAR8 *Description, >>>> IN EFI_STATUS Status >>>> ) >>>> { >>>> if (ConstraintAssertEnabled ()) { >>>> DEBUG ((EFI_D_ERROR, "\Constraint ASSERT (Status = %r): ", >>>> Status)); >>>> DebugAssert (FileName, LineNumber, Description) } >>>> >>>> return; >>>> } >>>> >>>> VOID >>>> IgnoreConstraintHandler ( >>>> IN CONST CHAR8 *FileName, >>>> IN UINTN LineNumber, IN CONST CHAR8 *Description, >>>> IN EFI_STATUS Status >>>> ) >>>> { >>>> return; >>>> } >>>> >>>> We could add macros for the code in the lib to call: >>>> >>>> #define CONSTRAINT_CHECK(Expression, Status) \ do { \ if >>>> (!(Expression)) { \ gActiveConstraintHandler (__FILE__, __LINE__, >>>> Expression, Status); \ return Status; \ } \ } while (FALSE) >>>> >>>> #define CONSTRAINT_REQUIRE(Expression, Status, Label) \ do { \ if >>>> (!(Expression)) { \ gActiveConstraintHandler (__FILE__, __LINE__, >>>> Expression, Status); \ goto Label; \ } \ } while (FALSE) >>>> >>>> >>>> As a caller we have now have control: >>>> EFI_STATUS Status; >>>> CHAR16 Dst[2]; >>>> >>>> SetConstraintHandler (IgnoreConstraintHandler); Status = StrCpyS >>>> (Dst, sizeof (Dst), L"Too Long"); Print (L"Dst =%s (%r)\n", Dst, >>>> Status); >>>> >>>> SetConstraintHandler (AssertConstraintHandler); StrCpyS (Dst, >>>> sizeof (Dst), L"Too Long"); >>>> >>>> Thanks, >>>> >>>> Andrew Fish >>>> >>>> PS Since I'm on a crazy idea roll another idea would be to add a >>>> MdePkgCompatability.dsc.inc file that could be used to future proof >>>> adding dependent libs to existing MdePkg libs. So a platform could >>>> include this .DSC and that would give them the default library >>>> mapping to keep code compiling. It will only work after other >>>> platforms start including it, but after that it would give default >>>> mappings for dependent libs. >>>> >>>> In our above example we could have added this and then existing >>>> builds that included MdePkgCompatability.dsc.inc would keep compiling. >>>> >>>> [LibraryClasses] >>>> >>>> DebugConstraintLib|MdePkg/Library/DebugConstraintLib/DebugConstrain >>>> DebugConstraintLib|tLib.inf >>>> >>>> >>>> >>>>> On Feb 15, 2020, at 11:38 AM, Michael D Kinney >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com><mailto:michael.d.kinney@intel. >>>>> com>> >>>>> wrote: >>>>> >>>>> Andrew, >>>>> I do not think of this as a globally scoped PCD.It can be set in >>>>> global scope in DSC.But is can also be set to values based on >>>>> module type or for specific modules.In the case of the safe string >>>>> functions, I think there is a desire to disable the constraint >>>>> asserts when building a UEFI App or UEFI Driver and implement >>>>> those modules to handle the error return values.Enabling the >>>>> constraint asserts for PEI Core, DXE Core, SMM/MM Core, PEIM, DXE, >>>>> SMM/MM modules makes sense to find incorrect input to these >>>>> functions from modules that can guarantee the inputs would never >>>>> return an error and catch these as part of dev/debug/validation builds. >>>>> I would not expect disabling on a module by module basis to be common. >>>>> I think the rule for API implementations is to only use >>>>> CONSTRAINT_ASSERT() for conditions that are also checked and >>>>> return an error or fail with predicable behavior that allows the >>>>> system to continue to function.ASSERT() is for conditions that the >>>>> systemcan notcontinue. >>>>> Best regards, >>>>> Mike >>>>> *From:*afish@apple.com >>>>> <mailto:afish@apple.com><mailto:afish@apple.com><afish@apple.com >>>>> <mailto:afish@apple.com> >>>>> <mailto:afish@apple.com>> >>>>> *Sent:*Friday, February 14, 2020 10:27 PM >>>>> *To:*vit9696 <vit9696@protonmail.com >>>>> <mailto:vit9696@protonmail.com><mailto:vit9696@protonmail.com>> >>>>> *Cc:*devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io><mailto:devel@edk2.groups.io>; >>>>> Kinney, Michael D <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>>; Gao, Liming >>>>> <liming.gao@intel.com >>>>> <mailto:liming.gao@intel.com><mailto:liming.gao@intel.com>>; Gao, >>>>> Zhichao <zhichao.gao@intel.com >>>>> <mailto:zhichao.gao@intel.com><mailto:zhichao.gao@intel.com>>; >>>>> Marvin Häuser >>>>> <marvin.haeuser@outlook.com >>>>> <mailto:marvin.haeuser@outlook.com><mailto:marvin.haeuser@outlook. >>>>> com>>; >>>>> Laszlo Ersek <lersek@redhat.com >>>>> <mailto:lersek@redhat.com><mailto:lersek@redhat.com>> >>>>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe >>>>> string constraint assertions Vitaly, Sorry after I sent the mail I >>>>> realized it may come across as me asking you to do work and that >>>>> was not my intent. >>>>> I will point out though that a non backward compatible change to >>>>> something as fundamental as the DebugLib is a very big deal. I've >>>>> got a few different custom implementations that would break with >>>>> this change as Mike proposed. Given breaking every one's debug lib >>>>> is such a big deal maybe it is something that we should do as a >>>>> long term plan vs. some incremental fix. So my intent was to start >>>>> a conversation about what else we might want to change if we are >>>>> going to break the world. The only think worse than breaking the >>>>> world is breaking the world frequently. >>>>> I'm also a little worried that we are taking things that are today >>>>> locally scoped like SAFE_STRING_CONSTRAINT_CHECK and >>>>> SAFE_PRINT_CONSTRAINT_CHECK and making them global constructs. I >>>>> think the way others have dealt with things like this is to make >>>>> them be DEBUG prints vs. ASSERTs. Also even something as simple as >>>>> SAFE_STRING_CONSTRAINT_CHECK could be called from code that wants >>>>> ASSERT and CONSTRAINT_ASSERT behavior. It is not clear to me that >>>>> the low level code knows the right thing to do in a global sense >>>>> even if there is a PCD. It almost seems like we should have >>>>> wrappers for the Safe string functions that implement the behavior >>>>> you want as a caller. I'm not sure about that, but it seems like >>>>> it is worth talking about? >>>>> Thanks, >>>>> Andrew Fish >>>>> >>>>> >>>>> On Feb 14, 2020, at 7:31 PM, vit9696 <vit9696@protonmail.com >>>>> <mailto:vit9696@protonmail.com> >>>>> <mailto:vit9696@protonmail.com>> wrote: >>>>> Hi Andrew, >>>>> While your suggestions look interesting, I am afraid they are >>>>> not >>>>> particularly what we want to cover with this discussion at the >>>>> moment. >>>>> Making assertions go through DEBUG printing functions/macros >>>>> is >>>>> what we have to strongly disagree about. Assertions and debug >>>>> prints are separate things configurable by separate PCDs. We >>>>> should not mix them. Introducing constraint assertions is a >>>>> logical step forward in understanding that different software >>>>> works in different environments. >>>>> >>>>> * There are normal, or, as I call them, invariant >>>>> assertions (e.g. preconditions), for places where the >>>>> function >>>>> cannot work unless the assertion is satisfied. This is >>>>> where >>>>> we ASSERT. >>>>> * There are constraint assertions, which signalise that bad >>>>> data >>>>> came through the function, even though the function was >>>>> called >>>>> from a trusted source. This is where we call CONSTRAINT_ASSERT. >>>>> >>>>> The thing we need is to have the latter separable >>>>> and configurable, because not every piece of software works in >>>>> a >>>>> trusted environment. Other than that, constraint assertions, >>>>> when >>>>> enabled, are not anyhow different from normal assertions in >>>>> the >>>>> sense of action taken. Assertions have configurable >>>>> breakpoints >>>>> and deadloops, and DEBUG prints go through a different route >>>>> in >>>>> DebugLib that may cause entirely different effects. For >>>>> example, >>>>> we halt execution upon printing to DEBUG_ERROR in our DebugLib >>>>> even in release builds. >>>>> >>>>> =To make it clear, currently I plan to add the following interface: >>>>> #define CONSTRAINT_ASSERT(Expression) \ >>>>> do { \ >>>>> if (DebugConstraintAssertEnabled ()) { \ >>>>> if (!(Expression)) { \ >>>>> _ASSERT (Expression); \ >>>>> ANALYZER_UNREACHABLE (); \ >>>>> } \ >>>>> } \ >>>>> } while (FALSE) >>>>> with DebugConstraintAssertEnabled implemented as >>>>> (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & >>>>> DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED | >>>>> DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED) == >>>>> DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED) >>>>> Your suggestion with require macros looks interesting indeed, >>>>> but >>>>> I believe it is in fact parallel to this discussion. The >>>>> change we >>>>> discuss introduces a new assertion primitive — constraint >>>>> assertions, while REQUIRE macros are mostly about advanced >>>>> syntax >>>>> sugar and higher level assertion primitives on top of existing >>>>> ones. Perhaps we can have this and make a good use of it, >>>>> especially given that it brought some practical benefit in >>>>> Apple, >>>>> but I would rather discuss this later once constraint >>>>> assertions >>>>> are merged into EDK II tree. >>>>> Best wishes, >>>>> Vitaly >>>>> On Sat, Feb 15, 2020 at 03:02, Andrew Fish <afish@apple.com >>>>> <mailto:afish@apple.com> >>>>> <mailto:afish@apple.com>> wrote: >>>>> >>>>> >>>>> >>>>> On Feb 14, 2020, at 2:50 PM, Michael D Kinney >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> wrote: >>>>> Hi Vitaly, >>>>> I agree that this proposal makes a lot of sense. We >>>>> recently added a new assert type called >>>>> STATIC_ASSERT() >>>>> for assert conditions that can be tested at build time. >>>>> A new assert type for checks that can be removed and >>>>> the >>>>> API still guarantees that it fails gracefully with a >>>>> proper return code is a good idea. Given we have >>>>> STATIC_ASSERT(), how about naming the new macro >>>>> CONSTRAINT_ASSERT()? >>>>> We also want the default to be enabled. The current >>>>> use of >>>>> bit 0x40 inPcdDebugPropertyMask is always clear, so we >>>>> want the asserts enabled when 0x40 is clear. We can >>>>> change >>>>> the name of the define bit to >>>>> DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED so bit 0x40 >>>>> needs >>>>> to be set inPcdDebugPropertyMaskto disable these types >>>>> of >>>>> asserts. >>>>> This approach does require all >>>>> theDebugLibimplementations >>>>> to be updated with the >>>>> newDebugConstraintAssertDisabled() >>>>> API. >>>>> >>>>> Mike, >>>>> If you wanted to be backward compatible you could just >>>>> use DebugAssertEnabled () but in place of _ASSERT() use >>>>> _CONSTRAINT_ASSERT >>>>> #define _ASSERT(Expression) DebugAssert (__FILE__, >>>>> __LINE__, >>>>> #Expression) >>>>> #define _CONSTRAINT_ASSERT(Expression) DebugPrint >>>>> (DEBUG_ERROR, "ASSERT %a(%d): %a\n",, __FILE__, __LINE__, >>>>> #Expression) >>>>> Not as elegant as the non backward compatible change, but >>>>> I >>>>> thought I'd throw it out there. >>>>> There are some ancient Apple C ASSERT macros >>>>> [AssertMacros.h] >>>>> that also have the concept of require. Require includes >>>>> an >>>>> exception label (goto label). It is like a >>>>> CONSTRAINT_ASSERT() >>>>> but with the label. On release builds the DEBUG prints are >>>>> skipped. >>>>> So we could do something like: >>>>> EFI_STATUS Status = EFI_INVALID_PARAMETER; >>>>> REQUIRE(Arg1 != NULL, ErrorExit); >>>>> REQUIRE(Arg2 != NULL, ErrorExit); >>>>> REQUIRE(Arg3 != NULL, ErrorExit); >>>>> ErrorExit: >>>>> return Status; >>>>> There is another form that allows an ACTION (a statement >>>>> to >>>>> execute. So you could have: >>>>> EFI_STATUS Status; >>>>> REQUIRE_ACTION(Arg1 != NULL, ErrorExit, Status = >>>>> EFI_INVALID_PARAMETER); >>>>> REQUIRE_ACTION(Arg2 != NULL, ErrorExit, Status = >>>>> EFI_INVALID_PARAMETER); >>>>> REQUIRE_ACTION(Arg3 != NULL, ErrorExit, Status = >>>>> EFI_INVALID_PARAMETER); >>>>> ErrorExit: >>>>> return Status; >>>>> If you use CONSTRAINT_ASSERT(); >>>>> if (Arg1 == NULL || Arg2 == NULL || Arg3 == NULL) { >>>>> CONSTRAINT_ASSERT (Arg1 != NULL); >>>>> CONSTRAINT_ASSERT (Arg2 != NULL); >>>>> CONSTRAINT_ASSERT (Arg3 != NULL); >>>>> return EFI_INVALID_PARAMETER; >>>>> } >>>>> I'd note error processing args on entry is the simplest case. >>>>> In a more complex case when cleanup is required the goto >>>>> label is more useful. >>>>> I guess we could argue for less typing and more symmetry >>>>> and >>>>> say use ASSERT, CONSTRAINT, and REQUIRE. I guess you could >>>>> add >>>>> an ASSERT_ACTION too. >>>>> The AssertMacros.h versions also support _quiet (skip the >>>>> print) and _string (add a string to the print) so you end >>>>> up >>>>> with: >>>>> REQUIRE >>>>> REQUIRE_STRING >>>>> REQUIRE_QUIET >>>>> REQUIRE_ACTION >>>>> REQUIRE_ACTION_STRING >>>>> REQUIRE_ACTION_QUIET >>>>> We could also end up with >>>>> CONSTRAINT >>>>> CONSTRAINT_STRING >>>>> CONSTRAINT_QUIET >>>>> I think the main idea behind _QUIET is you can silence >>>>> things >>>>> that are too noisy, and you can easily make noise things >>>>> show >>>>> up by removing the _QUIET to debug. >>>>> I'd thought I throw out the other forms for folks to think >>>>> about. I'm probably biased as I used to looking at code >>>>> and >>>>> seeing things like require_action_string(Arg1 != NULL, >>>>> ErrorExit, Status = EFI_INVALID_PARAMETER, "1st Arg1 >>>>> check"); >>>>> Thanks, >>>>> Andrew Fish >>>>> PS The old debug macros had 2 versions of CONSTRAINT check >>>>> and >>>>> verify. The check version was compiled out on a release >>>>> build, >>>>> the verify version always does the check and just skips >>>>> the >>>>> DEBUG print. >>>>> >>>>> Mike >>>>> *From:*vit9696 <vit9696@protonmail.com >>>>> <mailto:vit9696@protonmail.com> >>>>> <mailto:vit9696@protonmail.com>> >>>>> *Sent:*Friday, February 14, 2020 9:38 AM >>>>> *To:*Kinney, Michael D <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> >>>>> *Cc:*devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io><mailto:devel@edk2.groups.io>; >>>>> Gao, Liming <liming.gao@intel.com >>>>> <mailto:liming.gao@intel.com> >>>>> <mailto:liming.gao@intel.com>>; Gao, Zhichao >>>>> <zhichao.gao@intel.com >>>>> <mailto:zhichao.gao@intel.com><mailto:zhichao.gao@intel.com>>; >>>>> Marvin Häuser <marvin.haeuser@outlook.com >>>>> <mailto:marvin.haeuser@outlook.com> >>>>> <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek >>>>> <lersek@redhat.com >>>>> <mailto:lersek@redhat.com><mailto:lersek@redhat.com>> >>>>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>>> disable safe string constraint assertions >>>>> Michael, >>>>> Generalising the approach makes good sense to me, but >>>>> we >>>>> need to make an obvious distinguishable difference between: >>>>> - precondition and invariant assertions (i.e. >>>>> assertions, >>>>> where function will NOT work if they are violated) >>>>> - constraint asserts (i.e. assertions, which allow us >>>>> to >>>>> spot unintentional behaviour when parsing untrusted >>>>> data, >>>>> but which do not break function behaviour). >>>>> As we want to use this outside of SafeString, I >>>>> suggest >>>>> the following: >>>>> - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED >>>>> 0x40 >>>>> bit for PcdDebugPropertyMask instead >>>>> of PcdAssertOnSafeStringConstraints. >>>>> - Introduce DebugAssertConstraintEnabled DebugLib >>>>> function >>>>> to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. >>>>> - Introduce ASSERT_CONSTRAINT macro, that will assert >>>>> only >>>>> if DebugConstraintAssertEnabled returns true. >>>>> - Change SafeString ASSERTS >>>>> in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. >>>>> - Use ASSERT_CONSTRAINT in other places where necessary. >>>>> >>>>> I believe this way lines best with EDK II design. If >>>>> there >>>>> are no objections, I can submit the patch in the >>>>> beginning >>>>> of next week. >>>>> >>>>> Best wishes, >>>>> Vitaly >>>>> >>>>> 14 февр. 2020 г., в 20:00, Kinney, Michael D >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> написал(а): >>>>> Vitaly, >>>>> I want to make sure a feature PCD can be used to >>>>> disable ASSERT() behavior in more than just safe >>>>> string functions inBaseLib. >>>>> Can we consider changing the name and description >>>>> ofPcdAssertOnSafeStringConstraintsto be more >>>>> generic, >>>>> so if we find other lib APIs, the name will make sense? >>>>> Maybe something like:PcdEnableLibraryAssertChecks? >>>>> Default is TRUE. Can set to FALSE in DSC file to >>>>> disable ASSERT() checks. >>>>> Thanks, >>>>> Mike >>>>> *From:*devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io><devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io>>*On Behalf Of*Vitaly >>>>> Cheptsov via Groups.Io >>>>> *Sent:*Friday, February 14, 2020 3:55 AM >>>>> *To:*Kinney, Michael D <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>>; Gao, Liming >>>>> <liming.gao@intel.com >>>>> <mailto:liming.gao@intel.com><mailto:liming.gao@intel.com>>; >>>>> Gao, Zhichao <zhichao.gao@intel.com >>>>> <mailto:zhichao.gao@intel.com> >>>>> >>>>> <mailto:zhichao.gao@intel.com>>;devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io> >>>>> *Cc:*Marvin Häuser <marvin.haeuser@outlook.com >>>>> <mailto:marvin.haeuser@outlook.com> >>>>> <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek >>>>> <lersek@redhat.com >>>>> <mailto:lersek@redhat.com><mailto:lersek@redhat.com>> >>>>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD >>>>> to >>>>> disable safe string constraint assertions >>>>> Replying as per Liming's request for this to be >>>>> merged >>>>> into edk2-stable202002. >>>>> On Mon, Feb 10, 2020 at 14:12, vit9696 >>>>> <vit9696@protonmail.com >>>>> <mailto:vit9696@protonmail.com> >>>>> <mailto:vit9696@protonmail.com>> wrote: >>>>> >>>>> Hello, >>>>> >>>>> It has been quite some time since we submitted >>>>> the >>>>> patch with so far no negative response. As I >>>>> mentioned previously, my team will strongly >>>>> benefit from its landing in EDK II mainline. >>>>> Since >>>>> it does not add any regressions and can be >>>>> viewed >>>>> as a feature implementation for the rest of >>>>> EDK II >>>>> users, I request this to be merged upstream in >>>>> edk2-stable202002. >>>>> >>>>> Best wishes, >>>>> Vitaly >>>>> >>>>>> 27 янв. 2020 г., в 12:47, vit9696 >>>>> <vit9696@protonmail.com >>>>> <mailto:vit9696@protonmail.com> >>>>> <mailto:vit9696@protonmail.com>> написал(а): >>>>>> >>>>>> >>>>>> Hi Mike, >>>>>> >>>>>> Any progress with this? We would really benefit >>>>> from this landing in the next stable release. >>>>>> >>>>>> Best, >>>>>> Vitaly >>>>>> >>>>>>> 8 янв. 2020 г., в 19:35, Kinney, Michael D >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> написал(а): >>>>>>> >>>>>>> >>>>>>> Hi Vitaly, >>>>>>> >>>>>>> Thanks for the additional background. I would like a couple >>>>>>> extra day to review the PCD name and >>>>> the places >>>>>>> the PCD might potentially be used. >>>>>>> >>>>>>> If we find other APIs where ASSERT() behavior >>>>> is only >>>>>>> valuable during dev/debug to quickly identify >>>>> misuse >>>>>>> with trusted data and the API provides predicable return >>>>>>> behavior when ASSERT() is disabled, then >>>>> I would >>>>>>> like to have a pattern we can potentially apply >>>>> to all >>>>>>> these APIs across all packages. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Mike >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> >>>>> <mailto:devel@edk2.groups.io><devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io>> On >>>>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>>>> Sent: Monday, January 6, 2020 10:44 AM >>>>>>>> To: Kinney, Michael D >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> >>>>>>>> Cc:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io> >>>>>>>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add >>>>> PCD to >>>>>>>> disable safe string constraint assertions >>>>>>>> >>>>>>>> Hi Mike, >>>>>>>> >>>>>>>> Yes, the primary use case is for UEFI >>>>> Applications. We >>>>>>>> do not want to disable ASSERT’s completely, as assertions that >>>>>>>> make sense, i.e. the ones >>>>> signalising >>>>>>>> about interface misuse, are helpful for debugging. >>>>>>>> >>>>>>>> I have already explained in the BZ that >>>>> basically all >>>>>>>> safe string constraint assertions make no >>>>> sense for >>>>>>>> handling untrusted data. We find this use case >>>>> very >>>>>>>> logical, as these functions behave properly with assertions >>>>>>>> disabled and cover all these error conditions by the return >>>>>>>> statuses. In such >>>>> situation is >>>>>>>> not useful for these functions to assert, as >>>>> we end up >>>>>>>> inefficiently reimplementing the logic. I >>>>> would have >>>>>>>> liked the approach of discussing the interfaces individually, >>>>>>>> but I struggle to find any that >>>>> makes >>>>>>>> sense from this point of view. >>>>>>>> >>>>>>>> AsciiStrToGuid will ASSERT when the length of the passed string >>>>>>>> is odd. Functions that cannot, ahem, parse, for us are pretty >>>>>>>> much useless. >>>>>>>> AsciiStrCatS will ASSERT when the appended >>>>> string does >>>>>>>> not fit the buffer. For us this logic makes this function >>>>>>>> pretty much equivalent to deprecated >>>>> and thus >>>>>>>> unavailable AsciiStrCat, except it is also slower. >>>>>>>> >>>>>>>> My original suggestion was to remove the >>>>> assertions >>>>>>>> entirely, but several people here said that >>>>> they use >>>>>>>> them to verify usage errors when handling >>>>> trusted data. >>>>>>>> This makes good sense to me, so we suggest to >>>>> support >>>>>>>> both cases by introducing a PCD in this patch. >>>>>>>> >>>>>>>> Best wishes, >>>>>>>> Vitaly >>>>>>>> >>>>>>>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >>>>>>>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> написал(а): >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi Vitaly, >>>>>>>>> >>>>>>>>> Is the use case for UEFI Applications? >>>>>>>>> >>>>>>>>> There is a different mechanism to disable all >>>>>>>> ASSERT() >>>>>>>>> statements within a UEFI Application. >>>>>>>>> >>>>>>>>> If a component is consuming data from an >>>>> untrusted >>>>>>>> source, >>>>>>>>> then that component is required to verify the >>>>>>>> untrusted >>>>>>>>> data before passing it to a function that clearly >>>>>>>> documents >>>>>>>>> is input requirements. If this approach is >>>>> followed, >>>>>>>> then >>>>>>>>> the BaseLib functions can be used "as is" as >>>>> long as >>>>>>>> the >>>>>>>>> ASSERT() conditions are verified before calling. >>>>>>>>> >>>>>>>>> If there are some APIs that currently >>>>> document their >>>>>>>> ASSERT() >>>>>>>>> behavior and we think that ASSERT() behavior is >>>>>>>> incorrect and >>>>>>>>> should be handled by an existing error return >>>>> value, >>>>>>>> then we >>>>>>>>> should discuss each of those APIs individually. >>>>>>>>> >>>>>>>>> Mike >>>>>>>>> >>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> >>>>> <mailto:devel@edk2.groups.io><devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io>> On >>>>>>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>>>>>> Sent: Friday, January 3, 2020 9:13 AM To:devel@edk2.groups.io >>>>>>>>>> <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io> >>>>>>>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>>>>>> disable >>>>>>>>>> safe string constraint assertions >>>>>>>>>> >>>>>>>>>> REF: >>>>>>>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >>>>>>>>>> >>>>>>>>>> Requesting for merge in edk2-stable202002. >>>>>>>>>> >>>>>>>>>> Changes since V1: >>>>>>>>>> - Enable assertions by default to preserve the >>>>>>>> original >>>>>>>>>> behaviour >>>>>>>>>> - Fix bugzilla reference link >>>>>>>>>> - Update documentation in BaseLib.h >>>>>>>>>> >>>>>>>>>> Vitaly Cheptsov (1): >>>>>>>>>> MdePkg: Add PCD to disable safe string >>>>> constraint >>>>>>>>>> assertions >>>>>>>>>> >>>>>>>>>> MdePkg/MdePkg.dec | 6 ++ >>>>>>>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >>>>>>>>>> MdePkg/Include/Library/BaseLib.h | 74 >>>>>>>>>> +++++++++++++------- >>>>>>>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- MdePkg/MdePkg.uni >>>>>>>>>> | 6 ++ >>>>>>>>>> 5 files changed, 71 insertions(+), 30 >>>>> deletions(-) >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> 2.21.0 (Apple Git-122.2) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -=-=-=-=-=-= >>>>>>>>>> Groups.io <http://groups.io/><http://groups.io/>Links: You >>>>> receive all messages sent to >>>>>>>> this >>>>>>>>>> group. >>>>>>>>>> >>>>>>>>>> View/Reply Online (#52837): >>>>>>>>>> https://edk2.groups.io/g/devel/message/52837 >>>>>>>>>> Mute This Topic: >>>>>>>> https://groups.io/mt/69401948/1643496 >>>>>>>>>> Group Owner:devel+owner@edk2.groups.io >>>>>>>>>> <mailto:devel+owner@edk2.groups.io> >>>>> <mailto:devel+owner@edk2.groups.io> >>>>>>>>>> Unsubscribe:https://edk2.groups.io/g/devel/unsub >>>>>>>>>> [michael.d.kinney@intel.com >>>>>>>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>] >>>>>>>>>> -=-=-=-=-=-= >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>> >>> >> ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions 2020-02-20 10:18 ` Marvin Häuser 2020-03-03 19:38 ` Marvin Häuser @ 2020-03-03 20:27 ` Andrew Fish 1 sibling, 0 replies; 28+ messages in thread From: Andrew Fish @ 2020-03-03 20:27 UTC (permalink / raw) To: devel, mhaeuser Cc: Mike Kinney, vit9696, Gao, Liming, Gao, Zhichao, Marvin Häuser, Laszlo Ersek Marvin, If no one else objects to your proposal I don't want to block forward progress. My main intent was to have a conversation about this topic. > On Feb 20, 2020, at 2:18 AM, Marvin Häuser <mhaeuser@outlook.de> wrote: > > Hey Andrew, > > Thanks once again for your comments, mine are inline. > > Best regards, > Marvin > > Am 20.02.2020 um 00:55 schrieb Andrew Fish: >> >> >>> On Feb 17, 2020, at 12:26 AM, Marvin Häuser <mhaeuser@outlook.de >>> <mailto:mhaeuser@outlook.de>> wrote: >>> >>> Good day Andrew, >>> >>> First of all, thank you very much for putting this amount of thought >>> into the situation. I definitely agree with the problem you see, but I >>> could also live with Vitaly's proposal. However, I think you are >>> overcomplicating the situation a little. So, we do agree the caller >>> needs control of the behaviour, however your proposal needs "support" >>> from both the caller (choose the handler) and the callee (call the >>> handler) and is neither thread-safe nor event-safe as Vitaly already >>> pointed out. Fixing these problems would only worsen the complexity >>> situation in my opinion. >> >> Well EFI does not have Threads but it does have Events. I agree if an > > Sorry, we were refering to the UEFI Multi-Processor services, so threads > in a physical sense. I think very similar concerns apply regarding these > services as with software multithreading in this context, please correct > me if I'm wrong. It is not legal to do anything EFI on the APs. The locks in the DXE Core are TPL based so that is a software construct that only exists on the BSP. Technically speaking from a LibraryClass definition you should not run any libs that have ASSERT or DEBUG macros on the AP given the DebugLib Library Class is not MP safe. In reality it is possible to have an MP safe version of the DebugLib, and it is likely people are just getting away with MdePkg libs since it is not an issue unless they ASSERT or DEBUG print. I'd also point out that most code does not do work on the APs. On a typical x86 firmware it is Microcode, SMM, and the driver producing the MP Services than run code on the APs. While most other code does not. Note: EFI runs on the BSP, the other CPUs in the systems are called APs. Thanks, Andrew Fish > >> Event handler changes the constraint handler it would need to restore it >> so my proposal is missing an API. >> >> CONSTRAINT_HANDLER * >> GetConstraintHandler ( >> VOID >> ) >> { >> return gActiveConstraintHandler; >> } >> >> You an always use the standard EFI >> >> It is important to remember that all the EFI images >> (drivers/applications) are statically linked and they carry a unique >> instance of the Debug (or new Constraint) lib. So in your driver or App >> is very self contained. Also a lot of the events are phase related >> callbacks, and since there are no threads your drivers main thread is >> only really running when your driver, or app, dispatches. A lot of the >> callbacks are via protocols your driver publishes, but they have strict >> TPL rules so you can prevent reentrancy in general. So there is a lot >> more determinism in EFI vs. generic threaded coding. > > This is true, but yet all edge-cases should be covered (event > "interruption", actual (processor) interrupts, Multi-Processor > execution), so developers do not run into unexpected (seemingly) > undeterministic behaviour. However, I agree it is easier to ensure for > something like UEFI than for a full-fledged OS of course. > >> >>> >>> Diverging from both of your concepts entirely and keeping it very >>> simple, what keeps us from simply ASSERTing based on return value? We >>> could introduce a CONSTRAINT_ASSERT or similar, but it would be >>> completely different from Vitaly's proposal. It would be a caller macro >>> to ASSERT based on a pre-defined list of return statuses. >>> To achieve this, we would order all statuses by types (classes). At the >>> moment, I can only think of two: "environmental" and "constraint". For >>> example, "Out Of Resources" would be environmental and "Invalid >>> Parameter" would be constraint. We could define environment statuses >>> explicitly (as it is less likely new environment statuses are >>> introduced) and define constraint statuses as "not environmental" (to >>> silently support new constraint statuses, however of course it is a >>> little error-prone with forwards-compatibility). As a bonus, it forces >>> developers to use truly appropiate error codes and correctly propagate >>> them from callees for this to work. :) >>> This solution would be backwards-compatible in a compilation sense as it >>> can simply be a new macro, however for its possible complexity (switch >>> cases) I might actually prefer a function. But it would not be >>> backwards-compatible with the current functionality, which none of the >>> "caller-based" concepts can be anyway (the caller needs explicit code). >>> >> >> It is an interesting idea to add granularity, but at the end of the >> knowledge of the correct behavior of the lower level library code really >> lives in code that calls this. I will admit I can see some value to >> making RETURN_INVALID_PARAMETER different than RETURN_BUFFER_TOO_SMALL. > > So far, I prefer this idea as the only requirement is that function > contracts are well-designed (i.e. reflect environmental vs constraint > errors by using sensical return values). > >> >> I kind of went down the C11 path (thanks for mentioning the event issue) >> but there are other ways to empower the caller. > > It's definitely always a good idea to stick to standards and > conventions. It appears my proposal is not conventional at all. However, > I also believe there sometimes is a more pragmatic approach to things > without imposing significant disadvantages, especially by limiting the > scope. > > To be honest, I'm now interested for which cases the C11 path is > advantagous over my approach, except that setting the constraint handler > is a one-time thing if it remains consistent throughout execution. I'd > imagine for some more complex handling logic I cannot think of an > example of (all we really want to do is ASSERT conditionally afterall), > and I hope this is not something edk2 would ever run into - keep it > simple in my opinion. > >> >> We could let the caller decide the behavior. That implies passing >> CHECK_DEFAULT (PCD), CHECK_ASSERT, or CHECK_NULL. >> >> Then keep backward compatibility. >> #define StrCpyS(Destination,DestMax,Source)StrCpuSC(Destination, >> DestMax, Source, CHECK_DEFAULT) >> >> But that seems like a lot of thrash to the BaseLib and a lot of new >> magic to teach developers. > > Fully agreed with the last sentence. That was one of the approaches I > mentioned before my proposal, but I included it only for completeness' > sake so we have a full overview. This requires prototype adaption for > literally every function that may be used in such a way (which scopes > beyond just BaseLib), ugly macros and more. Please don't do that. :) > >> >> I would guess that the most common usage of my library would be to turn >> Constraint Checking off in the entire driver or app and then handle the >> errors manually. > > I'm afraid that's what most platforms will end up with. Our main concern > was we may not allow such constraint violations to ASSERT, that is our > main objective. However, if it can be made advantageous to future coding > practice, it would be nice to have a decent solution for the caller to > have some freedom. More about that below. > >> >> On driver/app entry do: >> >> SetConstraintHandler (IgnoreConstraintHandler); >> >> Then handle the errors you care about. >> >> Status = StrCpyS (Destination,DestMax,Source); >> if (EFI_ERROR (Status)) { >> ASSERT_EFI_ERROR (Status); >> return Status; >> } >> >> or >> >> Status = StrCpyS (Destination,DestMax,Source); >> if (Status == RETURN_INVALID_PARAMETER) { >> ASSERT_EFI_ERROR (Status); >> return Status; >> } >> >> At the end of the day I've code my driver and I know the rules I coded >> to and I want predictable behavior. > > Yes, agreed. We want predictable behaviour of "not ASSERTing" for > untrusted input, because we *know* it is untrusted and the function > *can* (and should be able to) handle it - it is simply not a precondition. > >> >> I can see a Si vendor developing with ASSERT Constraint off and then >> when the customer turns it on stuff starts randomly breaking. > > A lot of Si and core (DxeCore, PeiCore, related libraries) code seems to > ASSERT (frequently *only* ASSERT with no actual handling, especially for > AllocatePool() == NULL) when there are valid error situations possible, > so I hope this approach would be a handy tool to satisfy their needs > without this absolutely terrible practice. :) > > > > I think I understood all your comment individually, but I'm afraid I'm > not sure what your suggested solution is right now. You commented > semi-positively on my proposal, you did not revoke your C11 path, and > you furthermore introduced CHECK_* - this is brainstorming, and I think > it's a great thing, but I'm simply not sure what your exact goal or > prefered route is right now. > > My proposal is a bit unconventional and I agree it sounds hacky, but so > far I'm not convinced it would be bad practice at all. It boils down to > sensefully using the information a function returns. This requires > nothing but the function to return sensical information, which every > function should anyway. > > Your proposal is close to the C standard, and that is a good thing first > of all. However, with the rest of edk2 not being very compliant, I don't > think it's an instant win as in "we just comply to standards" because > this choice does not suddenly make porting existing code significantly > easier compared to the total lack of standard library support for > anything but Shell apps. > Pragmatically, I think it solves the same problem equally well, but at a > higher cost (ensuring the implementation is safe and, strictly speaking, > an ever-so-slight runtime overhead). > >> >>> However, CONSTRAINT_ASSERT might actually be too specific. Maybe we want >>> something like "ASSERT_RETURN" and have the classes "ASSERT_CONSTRAINT" >>> and "ASSERT_ENVIRONMENT" (bitfield, like with DEBUG). The reason for >>> this is embedded systems where the environment is trusted/known and the >>> firmware is supposed to be well-matched. Think of a SoC with soldered >>> RAM - the firmware can very well make assumption about memory capacity >>> (considering usage across all modules in the worst case) and might >>> actually want ASSERTs on something like "Out Of Resources" because it's >>> supposed to be impossible within the bounds of this specific design. >>> It's possible this would need a new PCD's involvement, for example to >>> tell DxeCore whether to ASSERT on environmental aspects too. There could >>> be an argument-less macro that uses PCD config as base, and an >>> argument-based macro that uses only the parameters. Of course this >>> cannot cover everyone's very specific preferences as it's a per-module >>> switch, but it's the best I can think of right now. >>> >>> Something a bit unrelated now, but it would make this easier. You >>> mentioned PeCoffLib as example, and I think it's a good one, however it >>> has its own return status type within the context[1] which I will most >>> definitely be getting rid of as part of my upcoming (in the far-ish >>> future :) ) RFC. Could we expand RETURN_STATUS to have (very high?) >>> reserved values for caller-defined error codes to have all RETURN_STATUS >>> macros apply to those special return values too? We'd need to define >>> them categorically as "constraint status", but I don't see how a library >>> would declare a new environment status anyway. >>> >>> Regarding MdePkgCompatability.dsc.inc, I think one can override library >>> classes from the inc by declaring them after the !include statement, >>> please correct me if I'm wrong. If so, I strongly agree and think it >>> should be the case for all packages, so one only overrides the defaults >>> when something specific (debugging, performance-optimized versions, ...) >>> is required - easier to read, easier to maintain. The content is >>> needed/there anyway as the libraries are declared in the package's own >>> DSC. >>> >> >> Yes it the new .inc DSC file could be overridden by the platform DSC >> that includes it. >> >> I think this is a more general idea than something for this given patch. >> It is something we could make sure we update every stable tag or so, but >> I guess someone has to go 1st :). It would make it easier on platforms >> when they update the edk2 version. > > Agreed. > >> >> >> >> >>> It would be nice if you had comments regarding every aspect I just >>> mentioned, it was just something coming to my mind this morning. Thanks >>> for the input so far, it's nice to see some movement now! >>> >> >> Sorry for the delay > > Sure, thanks for taking time to respond! > >> >> Thanks, >> >> Andrew Fish >> >>> Best regards, >>> Marvin >>> >>> [1] >>> https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L20-L31 >>> https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L159 >>> >>> Am 16.02.2020 um 22:25 schrieb Andrew Fish via Groups.Io: >>>> Mike, >>>> >>>> Sorry I don't think I totally understood what felt wrong to me, so I did >>>> a bad job explaining my concerns. Also I don't think I was thinking >>>> enough in the context of the C11 StdLib. >>>> >>>> I think my concern is still the global scope of the constraint, even if >>>> we tune it per module. For example the DXE Core can interact with >>>> PE/COFF images and other data that could have indirectly come from the >>>> disk. So conceptually you may want to ASSERT on some constraints and not >>>> on others. I think that is why I ratholed on expanding the error >>>> handling macros as that was more in the vein of having the caller deal >>>> with it, so that is kind of like what you would do pre C11. Also the >>>> DebugLib is probably one of the MdePkg Libs that has the most instances >>>> floating around, so I think we should change it in a non backwards way >>>> very carefully. >>>> >>>> So after reading up on the C11 implementation of Constraints I think my >>>> alternate proposal is for us to add a ConstraintLib modeled after C11 >>>> vs. updating the DebugLib. This would solve the 2 things that made me >>>> uncomfortable: 1) Extending the DebugLib API; 2) Giving the caller >>>> control of the ASSERT behavior. It would still have the down side of >>>> breaking builds as the BaseLib would get a new dependency, so we could >>>> talk about adding these functions to the DebugLib as the cost of >>>> replicating code. >>>> >>>> C11 defines constraint_handler_t and set_constraint_handler_s as a way >>>> for the caller to configure the behavior for bounds checked functions. I >>>> think that is the behavior I prefer. So if we are going to make a change >>>> that impacts DebugLib compatibility I just want to make sure we have a >>>> conversation about all the options. My primary goal is we have the >>>> conversation, and if folks don't agree with me that is fine at least we >>>> talked about it. >>>> >>>> What I'm thinking about is as simply exposing an API to control the >>>> Constraint handler like C11. This could be done via an ConstrainLib or >>>> extending the DebugLib. >>>> >>>> The basic implementation of the lib would look like: >>>> >>>> typedef >>>> VOID >>>> (EFIAPI *CONSTRAINT_HANDLER) ( >>>> IN CONST CHAR8 *FileName, >>>> IN UINTN LineNumber, >>>> IN CONST CHAR8 *Description, >>>> IN EFI_STATUS Status >>>> ); >>>> >>>> >>>> // Default to AssertConstraintHandler to make it easier to implement >>>> Base and XIP libs. >>>> // We could have a PCD that also sets the default handler in a Lib >>>> Constructor. The default handler is implementation defined in C11. >>>> CONSTRAINT_HANDLER gDefaultConstraintHandler = AssertConstraintHandler; >>>> CONSTRAINT_HANDLER gActiveConstraintHandler = gDefaultConstraintHandler; >>>> >>>> BOOLEAN >>>> EFIAPI >>>> ConstraintAssertEnabled ( >>>> VOID >>>> ) >>>> { >>>> return (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & >>>> DEBUG_PROPERTY_DEBUG_CONSTRAINT_ENABLED) != 0); >>>> } >>>> >>>> EFI_STATUS >>>> EFIAPI >>>> SetConstraintHandler ( >>>> IN CONSTRAINT_HANDLER Handler >>>> ) >>>> { >>>> if (Handler == NULL) { >>>> gActiveConstraintHandler = gDefaultConstraintHandler; >>>> } else { >>>> gActiveConstraintHandler = Handler; >>>> } >>>> } >>>> >>>> VOID >>>> AssertConstraintHandler ( >>>> IN CONST CHAR8 *FileName, >>>> IN UINTN LineNumber, >>>> IN CONST CHAR8 *Description, >>>> IN EFI_STATUS Status >>>> ) >>>> { >>>> if (ConstraintAssertEnabled ()) { >>>> DEBUG ((EFI_D_ERROR, "\Constraint ASSERT (Status = %r): ", Status)); >>>> DebugAssert (FileName, LineNumber, Description) >>>> } >>>> >>>> return; >>>> } >>>> >>>> VOID >>>> IgnoreConstraintHandler ( >>>> IN CONST CHAR8 *FileName, >>>> IN UINTN LineNumber, >>>> IN CONST CHAR8 *Description, >>>> IN EFI_STATUS Status >>>> ) >>>> { >>>> return; >>>> } >>>> >>>> We could add macros for the code in the lib to call: >>>> >>>> #define CONSTRAINT_CHECK(Expression, Status) \ >>>> do { \ >>>> if (!(Expression)) { \ >>>> gActiveConstraintHandler (__FILE__, __LINE__, Expression, Status); \ >>>> return Status; \ >>>> } \ >>>> } while (FALSE) >>>> >>>> #define CONSTRAINT_REQUIRE(Expression, Status, Label) \ >>>> do { \ >>>> if (!(Expression)) { \ >>>> gActiveConstraintHandler (__FILE__, __LINE__, Expression, Status); \ >>>> goto Label; \ >>>> } \ >>>> } while (FALSE) >>>> >>>> >>>> As a caller we have now have control: >>>> EFI_STATUS Status; >>>> CHAR16 Dst[2]; >>>> >>>> SetConstraintHandler (IgnoreConstraintHandler); >>>> Status = StrCpyS (Dst, sizeof (Dst), L"Too Long"); >>>> Print (L"Dst =%s (%r)\n", Dst, Status); >>>> >>>> SetConstraintHandler (AssertConstraintHandler); >>>> StrCpyS (Dst, sizeof (Dst), L"Too Long"); >>>> >>>> Thanks, >>>> >>>> Andrew Fish >>>> >>>> PS Since I'm on a crazy idea roll another idea would be to add a >>>> MdePkgCompatability.dsc.inc file that could be used to future proof >>>> adding dependent libs to existing MdePkg libs. So a platform could >>>> include this .DSC and that would give them the default library mapping >>>> to keep code compiling. It will only work after other platforms start >>>> including it, but after that it would give default mappings for >>>> dependent libs. >>>> >>>> In our above example we could have added this and then existing builds >>>> that included MdePkgCompatability.dsc.inc would keep compiling. >>>> >>>> [LibraryClasses] >>>> >>>> DebugConstraintLib|MdePkg/Library/DebugConstraintLib/DebugConstraintLib.inf >>>> >>>> >>>> >>>>> On Feb 15, 2020, at 11:38 AM, Michael D Kinney >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com><mailto:michael.d.kinney@intel.com>> >>>>> wrote: >>>>> >>>>> Andrew, >>>>> I do not think of this as a globally scoped PCD.It can be set in >>>>> global scope in DSC.But is can also be set to values based on module >>>>> type or for specific modules.In the case of the safe string functions, >>>>> I think there is a desire to disable the constraint asserts when >>>>> building a UEFI App or UEFI Driver and implement those modules to >>>>> handle the error return values.Enabling the constraint asserts for PEI >>>>> Core, DXE Core, SMM/MM Core, PEIM, DXE, SMM/MM modules makes sense to >>>>> find incorrect input to these functions from modules that can >>>>> guarantee the inputs would never return an error and catch these as >>>>> part of dev/debug/validation builds. >>>>> I would not expect disabling on a module by module basis to be common. >>>>> I think the rule for API implementations is to only use >>>>> CONSTRAINT_ASSERT() for conditions that are also checked and return an >>>>> error or fail with predicable behavior that allows the system to >>>>> continue to function.ASSERT() is for conditions that the systemcan >>>>> notcontinue. >>>>> Best regards, >>>>> Mike >>>>> *From:*afish@apple.com >>>>> <mailto:afish@apple.com><mailto:afish@apple.com><afish@apple.com >>>>> <mailto:afish@apple.com> >>>>> <mailto:afish@apple.com>> >>>>> *Sent:*Friday, February 14, 2020 10:27 PM >>>>> *To:*vit9696 <vit9696@protonmail.com >>>>> <mailto:vit9696@protonmail.com><mailto:vit9696@protonmail.com>> >>>>> *Cc:*devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io><mailto:devel@edk2.groups.io>; Kinney, >>>>> Michael D <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>>; Gao, Liming >>>>> <liming.gao@intel.com >>>>> <mailto:liming.gao@intel.com><mailto:liming.gao@intel.com>>; Gao, >>>>> Zhichao >>>>> <zhichao.gao@intel.com >>>>> <mailto:zhichao.gao@intel.com><mailto:zhichao.gao@intel.com>>; >>>>> Marvin Häuser >>>>> <marvin.haeuser@outlook.com >>>>> <mailto:marvin.haeuser@outlook.com><mailto:marvin.haeuser@outlook.com>>; >>>>> Laszlo Ersek <lersek@redhat.com >>>>> <mailto:lersek@redhat.com><mailto:lersek@redhat.com>> >>>>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe >>>>> string constraint assertions >>>>> Vitaly, >>>>> Sorry after I sent the mail I realized it may come across as me >>>>> asking you to do work and that was not my intent. >>>>> I will point out though that a non backward compatible change to >>>>> something as fundamental as the DebugLib is a very big deal. I've got >>>>> a few different custom implementations that would break with this >>>>> change as Mike proposed. Given breaking every one's debug lib is such >>>>> a big deal maybe it is something that we should do as a long term plan >>>>> vs. some incremental fix. So my intent was to start a conversation >>>>> about what else we might want to change if we are going to break the >>>>> world. The only think worse than breaking the world is breaking the >>>>> world frequently. >>>>> I'm also a little worried that we are taking things that are today >>>>> locally scoped like SAFE_STRING_CONSTRAINT_CHECK and >>>>> SAFE_PRINT_CONSTRAINT_CHECK and making them global constructs. I think >>>>> the way others have dealt with things like this is to make them be >>>>> DEBUG prints vs. ASSERTs. Also even something as simple as >>>>> SAFE_STRING_CONSTRAINT_CHECK could be called from code that wants >>>>> ASSERT and CONSTRAINT_ASSERT behavior. It is not clear to me that the >>>>> low level code knows the right thing to do in a global sense even if >>>>> there is a PCD. It almost seems like we should have wrappers for the >>>>> Safe string functions that implement the behavior you want as a >>>>> caller. I'm not sure about that, but it seems like it is worth talking >>>>> about? >>>>> Thanks, >>>>> Andrew Fish >>>>> >>>>> >>>>> On Feb 14, 2020, at 7:31 PM, vit9696 <vit9696@protonmail.com >>>>> <mailto:vit9696@protonmail.com> >>>>> <mailto:vit9696@protonmail.com>> wrote: >>>>> Hi Andrew, >>>>> While your suggestions look interesting, I am afraid they are not >>>>> particularly what we want to cover with this discussion at the >>>>> moment. >>>>> Making assertions go through DEBUG printing functions/macros is >>>>> what we have to strongly disagree about. Assertions and debug >>>>> prints are separate things configurable by separate PCDs. We >>>>> should not mix them. Introducing constraint assertions is a >>>>> logical step forward in understanding that different software >>>>> works in different environments. >>>>> >>>>> * There are normal, or, as I call them, invariant >>>>> assertions (e.g. preconditions), for places where the function >>>>> cannot work unless the assertion is satisfied. This is where >>>>> we ASSERT. >>>>> * There are constraint assertions, which signalise that bad data >>>>> came through the function, even though the function was called >>>>> from a trusted source. This is where we call CONSTRAINT_ASSERT. >>>>> >>>>> The thing we need is to have the latter separable >>>>> and configurable, because not every piece of software works in a >>>>> trusted environment. Other than that, constraint assertions, when >>>>> enabled, are not anyhow different from normal assertions in the >>>>> sense of action taken. Assertions have configurable breakpoints >>>>> and deadloops, and DEBUG prints go through a different route in >>>>> DebugLib that may cause entirely different effects. For example, >>>>> we halt execution upon printing to DEBUG_ERROR in our DebugLib >>>>> even in release builds. >>>>> >>>>> =To make it clear, currently I plan to add the following interface: >>>>> #define CONSTRAINT_ASSERT(Expression) \ >>>>> do { \ >>>>> if (DebugConstraintAssertEnabled ()) { \ >>>>> if (!(Expression)) { \ >>>>> _ASSERT (Expression); \ >>>>> ANALYZER_UNREACHABLE (); \ >>>>> } \ >>>>> } \ >>>>> } while (FALSE) >>>>> with DebugConstraintAssertEnabled implemented as >>>>> (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & >>>>> DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED | >>>>> DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED) == >>>>> DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED) >>>>> Your suggestion with require macros looks interesting indeed, but >>>>> I believe it is in fact parallel to this discussion. The change we >>>>> discuss introduces a new assertion primitive — constraint >>>>> assertions, while REQUIRE macros are mostly about advanced syntax >>>>> sugar and higher level assertion primitives on top of existing >>>>> ones. Perhaps we can have this and make a good use of it, >>>>> especially given that it brought some practical benefit in Apple, >>>>> but I would rather discuss this later once constraint assertions >>>>> are merged into EDK II tree. >>>>> Best wishes, >>>>> Vitaly >>>>> On Sat, Feb 15, 2020 at 03:02, Andrew Fish <afish@apple.com >>>>> <mailto:afish@apple.com> >>>>> <mailto:afish@apple.com>> wrote: >>>>> >>>>> >>>>> >>>>> On Feb 14, 2020, at 2:50 PM, Michael D Kinney >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> wrote: >>>>> Hi Vitaly, >>>>> I agree that this proposal makes a lot of sense. We >>>>> recently added a new assert type called STATIC_ASSERT() >>>>> for assert conditions that can be tested at build time. >>>>> A new assert type for checks that can be removed and the >>>>> API still guarantees that it fails gracefully with a >>>>> proper return code is a good idea. Given we have >>>>> STATIC_ASSERT(), how about naming the new macro >>>>> CONSTRAINT_ASSERT()? >>>>> We also want the default to be enabled. The current use of >>>>> bit 0x40 inPcdDebugPropertyMask is always clear, so we >>>>> want the asserts enabled when 0x40 is clear. We can change >>>>> the name of the define bit to >>>>> DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED so bit 0x40 needs >>>>> to be set inPcdDebugPropertyMaskto disable these types of >>>>> asserts. >>>>> This approach does require all theDebugLibimplementations >>>>> to be updated with the newDebugConstraintAssertDisabled() >>>>> API. >>>>> >>>>> Mike, >>>>> If you wanted to be backward compatible you could just >>>>> use DebugAssertEnabled () but in place of _ASSERT() use >>>>> _CONSTRAINT_ASSERT >>>>> #define _ASSERT(Expression) DebugAssert (__FILE__, __LINE__, >>>>> #Expression) >>>>> #define _CONSTRAINT_ASSERT(Expression) DebugPrint >>>>> (DEBUG_ERROR, "ASSERT %a(%d): %a\n",, __FILE__, __LINE__, >>>>> #Expression) >>>>> Not as elegant as the non backward compatible change, but I >>>>> thought I'd throw it out there. >>>>> There are some ancient Apple C ASSERT macros [AssertMacros.h] >>>>> that also have the concept of require. Require includes an >>>>> exception label (goto label). It is like a CONSTRAINT_ASSERT() >>>>> but with the label. On release builds the DEBUG prints are >>>>> skipped. >>>>> So we could do something like: >>>>> EFI_STATUS Status = EFI_INVALID_PARAMETER; >>>>> REQUIRE(Arg1 != NULL, ErrorExit); >>>>> REQUIRE(Arg2 != NULL, ErrorExit); >>>>> REQUIRE(Arg3 != NULL, ErrorExit); >>>>> ErrorExit: >>>>> return Status; >>>>> There is another form that allows an ACTION (a statement to >>>>> execute. So you could have: >>>>> EFI_STATUS Status; >>>>> REQUIRE_ACTION(Arg1 != NULL, ErrorExit, Status = >>>>> EFI_INVALID_PARAMETER); >>>>> REQUIRE_ACTION(Arg2 != NULL, ErrorExit, Status = >>>>> EFI_INVALID_PARAMETER); >>>>> REQUIRE_ACTION(Arg3 != NULL, ErrorExit, Status = >>>>> EFI_INVALID_PARAMETER); >>>>> ErrorExit: >>>>> return Status; >>>>> If you use CONSTRAINT_ASSERT(); >>>>> if (Arg1 == NULL || Arg2 == NULL || Arg3 == NULL) { >>>>> CONSTRAINT_ASSERT (Arg1 != NULL); >>>>> CONSTRAINT_ASSERT (Arg2 != NULL); >>>>> CONSTRAINT_ASSERT (Arg3 != NULL); >>>>> return EFI_INVALID_PARAMETER; >>>>> } >>>>> I'd note error processing args on entry is the simplest case. >>>>> In a more complex case when cleanup is required the goto >>>>> label is more useful. >>>>> I guess we could argue for less typing and more symmetry and >>>>> say use ASSERT, CONSTRAINT, and REQUIRE. I guess you could add >>>>> an ASSERT_ACTION too. >>>>> The AssertMacros.h versions also support _quiet (skip the >>>>> print) and _string (add a string to the print) so you end up >>>>> with: >>>>> REQUIRE >>>>> REQUIRE_STRING >>>>> REQUIRE_QUIET >>>>> REQUIRE_ACTION >>>>> REQUIRE_ACTION_STRING >>>>> REQUIRE_ACTION_QUIET >>>>> We could also end up with >>>>> CONSTRAINT >>>>> CONSTRAINT_STRING >>>>> CONSTRAINT_QUIET >>>>> I think the main idea behind _QUIET is you can silence things >>>>> that are too noisy, and you can easily make noise things show >>>>> up by removing the _QUIET to debug. >>>>> I'd thought I throw out the other forms for folks to think >>>>> about. I'm probably biased as I used to looking at code and >>>>> seeing things like require_action_string(Arg1 != NULL, >>>>> ErrorExit, Status = EFI_INVALID_PARAMETER, "1st Arg1 check"); >>>>> Thanks, >>>>> Andrew Fish >>>>> PS The old debug macros had 2 versions of CONSTRAINT check and >>>>> verify. The check version was compiled out on a release build, >>>>> the verify version always does the check and just skips the >>>>> DEBUG print. >>>>> >>>>> Mike >>>>> *From:*vit9696 <vit9696@protonmail.com >>>>> <mailto:vit9696@protonmail.com> >>>>> <mailto:vit9696@protonmail.com>> >>>>> *Sent:*Friday, February 14, 2020 9:38 AM >>>>> *To:*Kinney, Michael D <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> >>>>> *Cc:*devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io><mailto:devel@edk2.groups.io>; >>>>> Gao, Liming <liming.gao@intel.com >>>>> <mailto:liming.gao@intel.com> >>>>> <mailto:liming.gao@intel.com>>; Gao, Zhichao >>>>> <zhichao.gao@intel.com >>>>> <mailto:zhichao.gao@intel.com><mailto:zhichao.gao@intel.com>>; >>>>> Marvin Häuser <marvin.haeuser@outlook.com >>>>> <mailto:marvin.haeuser@outlook.com> >>>>> <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek >>>>> <lersek@redhat.com >>>>> <mailto:lersek@redhat.com><mailto:lersek@redhat.com>> >>>>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>>> disable safe string constraint assertions >>>>> Michael, >>>>> Generalising the approach makes good sense to me, but we >>>>> need to make an obvious distinguishable difference between: >>>>> - precondition and invariant assertions (i.e. assertions, >>>>> where function will NOT work if they are violated) >>>>> - constraint asserts (i.e. assertions, which allow us to >>>>> spot unintentional behaviour when parsing untrusted data, >>>>> but which do not break function behaviour). >>>>> As we want to use this outside of SafeString, I suggest >>>>> the following: >>>>> - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED 0x40 >>>>> bit for PcdDebugPropertyMask instead >>>>> of PcdAssertOnSafeStringConstraints. >>>>> - Introduce DebugAssertConstraintEnabled DebugLib function >>>>> to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. >>>>> - Introduce ASSERT_CONSTRAINT macro, that will assert only >>>>> if DebugConstraintAssertEnabled returns true. >>>>> - Change SafeString ASSERTS >>>>> in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. >>>>> - Use ASSERT_CONSTRAINT in other places where necessary. >>>>> >>>>> I believe this way lines best with EDK II design. If there >>>>> are no objections, I can submit the patch in the beginning >>>>> of next week. >>>>> >>>>> Best wishes, >>>>> Vitaly >>>>> >>>>> 14 февр. 2020 г., в 20:00, Kinney, Michael D >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> написал(а): >>>>> Vitaly, >>>>> I want to make sure a feature PCD can be used to >>>>> disable ASSERT() behavior in more than just safe >>>>> string functions inBaseLib. >>>>> Can we consider changing the name and description >>>>> ofPcdAssertOnSafeStringConstraintsto be more generic, >>>>> so if we find other lib APIs, the name will make sense? >>>>> Maybe something like:PcdEnableLibraryAssertChecks? >>>>> Default is TRUE. Can set to FALSE in DSC file to >>>>> disable ASSERT() checks. >>>>> Thanks, >>>>> Mike >>>>> *From:*devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io><devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io>>*On Behalf Of*Vitaly >>>>> Cheptsov via Groups.Io >>>>> *Sent:*Friday, February 14, 2020 3:55 AM >>>>> *To:*Kinney, Michael D <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>>; Gao, Liming >>>>> <liming.gao@intel.com >>>>> <mailto:liming.gao@intel.com><mailto:liming.gao@intel.com>>; >>>>> Gao, Zhichao <zhichao.gao@intel.com >>>>> <mailto:zhichao.gao@intel.com> >>>>> <mailto:zhichao.gao@intel.com>>;devel@edk2.groups.io >>>>> <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io> >>>>> *Cc:*Marvin Häuser <marvin.haeuser@outlook.com >>>>> <mailto:marvin.haeuser@outlook.com> >>>>> <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek >>>>> <lersek@redhat.com >>>>> <mailto:lersek@redhat.com><mailto:lersek@redhat.com>> >>>>> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>>> disable safe string constraint assertions >>>>> Replying as per Liming's request for this to be merged >>>>> into edk2-stable202002. >>>>> On Mon, Feb 10, 2020 at 14:12, vit9696 >>>>> <vit9696@protonmail.com <mailto:vit9696@protonmail.com> >>>>> <mailto:vit9696@protonmail.com>> wrote: >>>>> >>>>> Hello, >>>>> >>>>> It has been quite some time since we submitted the >>>>> patch with so far no negative response. As I >>>>> mentioned previously, my team will strongly >>>>> benefit from its landing in EDK II mainline. Since >>>>> it does not add any regressions and can be viewed >>>>> as a feature implementation for the rest of EDK II >>>>> users, I request this to be merged upstream in >>>>> edk2-stable202002. >>>>> >>>>> Best wishes, >>>>> Vitaly >>>>> >>>>>> 27 янв. 2020 г., в 12:47, vit9696 >>>>> <vit9696@protonmail.com >>>>> <mailto:vit9696@protonmail.com> >>>>> <mailto:vit9696@protonmail.com>> написал(а): >>>>>> >>>>>> >>>>>> Hi Mike, >>>>>> >>>>>> Any progress with this? We would really benefit >>>>> from this landing in the next stable release. >>>>>> >>>>>> Best, >>>>>> Vitaly >>>>>> >>>>>>> 8 янв. 2020 г., в 19:35, Kinney, Michael D >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> написал(а): >>>>>>> >>>>>>> >>>>>>> Hi Vitaly, >>>>>>> >>>>>>> Thanks for the additional background. I would like >>>>>>> a couple extra day to review the PCD name and >>>>> the places >>>>>>> the PCD might potentially be used. >>>>>>> >>>>>>> If we find other APIs where ASSERT() behavior >>>>> is only >>>>>>> valuable during dev/debug to quickly identify >>>>> misuse >>>>>>> with trusted data and the API provides predicable >>>>>>> return behavior when ASSERT() is disabled, then >>>>> I would >>>>>>> like to have a pattern we can potentially apply >>>>> to all >>>>>>> these APIs across all packages. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Mike >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io><devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io>> On >>>>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>>>> Sent: Monday, January 6, 2020 10:44 AM >>>>>>>> To: Kinney, Michael D >>>>> <michael.d.kinney@intel.com >>>>> <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> >>>>>>>> Cc:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io> >>>>>>>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add >>>>> PCD to >>>>>>>> disable safe string constraint assertions >>>>>>>> >>>>>>>> Hi Mike, >>>>>>>> >>>>>>>> Yes, the primary use case is for UEFI >>>>> Applications. We >>>>>>>> do not want to disable ASSERT’s completely, as >>>>>>>> assertions that make sense, i.e. the ones >>>>> signalising >>>>>>>> about interface misuse, are helpful for debugging. >>>>>>>> >>>>>>>> I have already explained in the BZ that >>>>> basically all >>>>>>>> safe string constraint assertions make no >>>>> sense for >>>>>>>> handling untrusted data. We find this use case >>>>> very >>>>>>>> logical, as these functions behave properly with >>>>>>>> assertions disabled and cover all these error >>>>>>>> conditions by the return statuses. In such >>>>> situation is >>>>>>>> not useful for these functions to assert, as >>>>> we end up >>>>>>>> inefficiently reimplementing the logic. I >>>>> would have >>>>>>>> liked the approach of discussing the interfaces >>>>>>>> individually, but I struggle to find any that >>>>> makes >>>>>>>> sense from this point of view. >>>>>>>> >>>>>>>> AsciiStrToGuid will ASSERT when the length of the >>>>>>>> passed string is odd. Functions that cannot, ahem, >>>>>>>> parse, for us are pretty much useless. >>>>>>>> AsciiStrCatS will ASSERT when the appended >>>>> string does >>>>>>>> not fit the buffer. For us this logic makes this >>>>>>>> function pretty much equivalent to deprecated >>>>> and thus >>>>>>>> unavailable AsciiStrCat, except it is also slower. >>>>>>>> >>>>>>>> My original suggestion was to remove the >>>>> assertions >>>>>>>> entirely, but several people here said that >>>>> they use >>>>>>>> them to verify usage errors when handling >>>>> trusted data. >>>>>>>> This makes good sense to me, so we suggest to >>>>> support >>>>>>>> both cases by introducing a PCD in this patch. >>>>>>>> >>>>>>>> Best wishes, >>>>>>>> Vitaly >>>>>>>> >>>>>>>>> 6 янв. 2020 г., в 21:28, Kinney, Michael D >>>>>>>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>> написал(а): >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi Vitaly, >>>>>>>>> >>>>>>>>> Is the use case for UEFI Applications? >>>>>>>>> >>>>>>>>> There is a different mechanism to disable all >>>>>>>> ASSERT() >>>>>>>>> statements within a UEFI Application. >>>>>>>>> >>>>>>>>> If a component is consuming data from an >>>>> untrusted >>>>>>>> source, >>>>>>>>> then that component is required to verify the >>>>>>>> untrusted >>>>>>>>> data before passing it to a function that clearly >>>>>>>> documents >>>>>>>>> is input requirements. If this approach is >>>>> followed, >>>>>>>> then >>>>>>>>> the BaseLib functions can be used "as is" as >>>>> long as >>>>>>>> the >>>>>>>>> ASSERT() conditions are verified before calling. >>>>>>>>> >>>>>>>>> If there are some APIs that currently >>>>> document their >>>>>>>> ASSERT() >>>>>>>>> behavior and we think that ASSERT() behavior is >>>>>>>> incorrect and >>>>>>>>> should be handled by an existing error return >>>>> value, >>>>>>>> then we >>>>>>>>> should discuss each of those APIs individually. >>>>>>>>> >>>>>>>>> Mike >>>>>>>>> >>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io><devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io>> On >>>>>>>>>> Behalf Of Vitaly Cheptsov via Groups.Io >>>>>>>>>> Sent: Friday, January 3, 2020 9:13 AM >>>>>>>>>> To:devel@edk2.groups.io <mailto:devel@edk2.groups.io> >>>>> <mailto:devel@edk2.groups.io> >>>>>>>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >>>>>>>> disable >>>>>>>>>> safe string constraint assertions >>>>>>>>>> >>>>>>>>>> REF: >>>>>>>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >>>>>>>>>> >>>>>>>>>> Requesting for merge in edk2-stable202002. >>>>>>>>>> >>>>>>>>>> Changes since V1: >>>>>>>>>> - Enable assertions by default to preserve the >>>>>>>> original >>>>>>>>>> behaviour >>>>>>>>>> - Fix bugzilla reference link >>>>>>>>>> - Update documentation in BaseLib.h >>>>>>>>>> >>>>>>>>>> Vitaly Cheptsov (1): >>>>>>>>>> MdePkg: Add PCD to disable safe string >>>>> constraint >>>>>>>>>> assertions >>>>>>>>>> >>>>>>>>>> MdePkg/MdePkg.dec | 6 ++ >>>>>>>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >>>>>>>>>> MdePkg/Include/Library/BaseLib.h | 74 >>>>>>>>>> +++++++++++++------- >>>>>>>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >>>>>>>>>> MdePkg/MdePkg.uni | 6 ++ >>>>>>>>>> 5 files changed, 71 insertions(+), 30 >>>>> deletions(-) >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> 2.21.0 (Apple Git-122.2) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -=-=-=-=-=-= >>>>>>>>>> Groups.io <http://groups.io/><http://groups.io/>Links: You >>>>> receive all messages sent to >>>>>>>> this >>>>>>>>>> group. >>>>>>>>>> >>>>>>>>>> View/Reply Online (#52837): >>>>>>>>>> https://edk2.groups.io/g/devel/message/52837 >>>>>>>>>> Mute This Topic: >>>>>>>> https://groups.io/mt/69401948/1643496 >>>>>>>>>> Group Owner:devel+owner@edk2.groups.io >>>>>>>>>> <mailto:devel+owner@edk2.groups.io> >>>>> <mailto:devel+owner@edk2.groups.io> >>>>>>>>>> Unsubscribe:https://edk2.groups.io/g/devel/unsub >>>>>>>>>> [michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >>>>> <mailto:michael.d.kinney@intel.com>] >>>>>>>>>> -=-=-=-=-=-= >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>> >>> >> > > > ^ permalink raw reply [flat|nested] 28+ messages in thread
[parent not found: <15F4232304E080CF.5373@groups.io>]
* Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions [not found] ` <15F4232304E080CF.5373@groups.io> @ 2020-02-17 9:36 ` Marvin Häuser 0 siblings, 0 replies; 28+ messages in thread From: Marvin Häuser @ 2020-02-17 9:36 UTC (permalink / raw) To: devel@edk2.groups.io, Marvin Häuser, afish@apple.com, Mike Kinney Cc: vit9696, Gao, Liming, Gao, Zhichao, Laszlo Ersek [-- Attachment #1: Type: text/plain, Size: 38138 bytes --] Sorry, because of private feedback I just want to quickly clarify this has absolutely no effect on the current ASSERT macro and concept, it is not considered deprecated and existing ASSERTs (most of them) would remain as-is. This is basically an extended ASSERT_RETURN_ERROR. Best regards, Marvin Von: Marvin Häuser <mhaeuser@outlook.de> Gesendet: Montag, 17. Februar 2020 09:26 An: devel@edk2.groups.io; afish@apple.com; Mike Kinney Cc: vit9696; Gao, Liming; Gao, Zhichao; Marvin Häuser; Laszlo Ersek Betreff: Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe string constraint assertions Good day Andrew, First of all, thank you very much for putting this amount of thought into the situation. I definitely agree with the problem you see, but I could also live with Vitaly's proposal. However, I think you are overcomplicating the situation a little. So, we do agree the caller needs control of the behaviour, however your proposal needs "support" from both the caller (choose the handler) and the callee (call the handler) and is neither thread-safe nor event-safe as Vitaly already pointed out. Fixing these problems would only worsen the complexity situation in my opinion. Diverging from both of your concepts entirely and keeping it very simple, what keeps us from simply ASSERTing based on return value? We could introduce a CONSTRAINT_ASSERT or similar, but it would be completely different from Vitaly's proposal. It would be a caller macro to ASSERT based on a pre-defined list of return statuses. To achieve this, we would order all statuses by types (classes). At the moment, I can only think of two: "environmental" and "constraint". For example, "Out Of Resources" would be environmental and "Invalid Parameter" would be constraint. We could define environment statuses explicitly (as it is less likely new environment statuses are introduced) and define constraint statuses as "not environmental" (to silently support new constraint statuses, however of course it is a little error-prone with forwards-compatibility). As a bonus, it forces developers to use truly appropiate error codes and correctly propagate them from callees for this to work. :) This solution would be backwards-compatible in a compilation sense as it can simply be a new macro, however for its possible complexity (switch cases) I might actually prefer a function. But it would not be backwards-compatible with the current functionality, which none of the "caller-based" concepts can be anyway (the caller needs explicit code). However, CONSTRAINT_ASSERT might actually be too specific. Maybe we want something like "ASSERT_RETURN" and have the classes "ASSERT_CONSTRAINT" and "ASSERT_ENVIRONMENT" (bitfield, like with DEBUG). The reason for this is embedded systems where the environment is trusted/known and the firmware is supposed to be well-matched. Think of a SoC with soldered RAM - the firmware can very well make assumption about memory capacity (considering usage across all modules in the worst case) and might actually want ASSERTs on something like "Out Of Resources" because it's supposed to be impossible within the bounds of this specific design. It's possible this would need a new PCD's involvement, for example to tell DxeCore whether to ASSERT on environmental aspects too. There could be an argument-less macro that uses PCD config as base, and an argument-based macro that uses only the parameters. Of course this cannot cover everyone's very specific preferences as it's a per-module switch, but it's the best I can think of right now. Something a bit unrelated now, but it would make this easier. You mentioned PeCoffLib as example, and I think it's a good one, however it has its own return status type within the context[1] which I will most definitely be getting rid of as part of my upcoming (in the far-ish future :) ) RFC. Could we expand RETURN_STATUS to have (very high?) reserved values for caller-defined error codes to have all RETURN_STATUS macros apply to those special return values too? We'd need to define them categorically as "constraint status", but I don't see how a library would declare a new environment status anyway. Regarding MdePkgCompatability.dsc.inc, I think one can override library classes from the inc by declaring them after the !include statement, please correct me if I'm wrong. If so, I strongly agree and think it should be the case for all packages, so one only overrides the defaults when something specific (debugging, performance-optimized versions, ...) is required - easier to read, easier to maintain. The content is needed/there anyway as the libraries are declared in the package's own DSC. It would be nice if you had comments regarding every aspect I just mentioned, it was just something coming to my mind this morning. Thanks for the input so far, it's nice to see some movement now! Best regards, Marvin [1] https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L20-L31 https://github.com/tianocore/edk2/blob/f1d78c489a39971b5aac5d2fc8a39bfa925c3c5d/MdePkg/Include/Library/PeCoffLib.h#L159 Am 16.02.2020 um 22:25 schrieb Andrew Fish via Groups.Io: > Mike, > > Sorry I don't think I totally understood what felt wrong to me, so I did > a bad job explaining my concerns. Also I don't think I was thinking > enough in the context of the C11 StdLib. > > I think my concern is still the global scope of the constraint, even if > we tune it per module. For example the DXE Core can interact with > PE/COFF images and other data that could have indirectly come from the > disk. So conceptually you may want to ASSERT on some constraints and not > on others. I think that is why I ratholed on expanding the error > handling macros as that was more in the vein of having the caller deal > with it, so that is kind of like what you would do pre C11. Also the > DebugLib is probably one of the MdePkg Libs that has the most instances > floating around, so I think we should change it in a non backwards way > very carefully. > > So after reading up on the C11 implementation of Constraints I think my > alternate proposal is for us to add a ConstraintLib modeled after C11 > vs. updating the DebugLib. This would solve the 2 things that made me > uncomfortable: 1) Extending the DebugLib API; 2) Giving the caller > control of the ASSERT behavior. It would still have the down side of > breaking builds as the BaseLib would get a new dependency, so we could > talk about adding these functions to the DebugLib as the cost of > replicating code. > > C11 defines constraint_handler_t and set_constraint_handler_s as a way > for the caller to configure the behavior for bounds checked functions. I > think that is the behavior I prefer. So if we are going to make a change > that impacts DebugLib compatibility I just want to make sure we have a > conversation about all the options. My primary goal is we have the > conversation, and if folks don't agree with me that is fine at least we > talked about it. > > What I'm thinking about is as simply exposing an API to control the > Constraint handler like C11. This could be done via an ConstrainLib or > extending the DebugLib. > > The basic implementation of the lib would look like: > > typedef > VOID > (EFIAPI *CONSTRAINT_HANDLER) ( > IN CONST CHAR8 *FileName, > IN UINTN LineNumber, > IN CONST CHAR8 *Description, > IN EFI_STATUS Status > ); > > > // Default to AssertConstraintHandler to make it easier to implement > Base and XIP libs. > // We could have a PCD that also sets the default handler in a Lib > Constructor. The default handler is implementation defined in C11. > CONSTRAINT_HANDLER gDefaultConstraintHandler = AssertConstraintHandler; > CONSTRAINT_HANDLER gActiveConstraintHandler = gDefaultConstraintHandler; > > BOOLEAN > EFIAPI > ConstraintAssertEnabled ( > VOID > ) > { > return (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & > DEBUG_PROPERTY_DEBUG_CONSTRAINT_ENABLED) != 0); > } > > EFI_STATUS > EFIAPI > SetConstraintHandler ( > IN CONSTRAINT_HANDLER Handler > ) > { > if (Handler == NULL) { > gActiveConstraintHandler = gDefaultConstraintHandler; > } else { > gActiveConstraintHandler = Handler; > } > } > > VOID > AssertConstraintHandler ( > IN CONST CHAR8 *FileName, > IN UINTN LineNumber, > IN CONST CHAR8 *Description, > IN EFI_STATUS Status > ) > { > if (ConstraintAssertEnabled ()) { > DEBUG ((EFI_D_ERROR, "\Constraint ASSERT (Status = %r): ", Status)); > DebugAssert (FileName, LineNumber, Description) > } > > return; > } > > VOID > IgnoreConstraintHandler ( > IN CONST CHAR8 *FileName, > IN UINTN LineNumber, > IN CONST CHAR8 *Description, > IN EFI_STATUS Status > ) > { > return; > } > > We could add macros for the code in the lib to call: > > #define CONSTRAINT_CHECK(Expression, Status) \ > do { \ > if (!(Expression)) { \ > gActiveConstraintHandler (__FILE__, __LINE__, Expression, Status); \ > return Status; \ > } \ > } while (FALSE) > > #define CONSTRAINT_REQUIRE(Expression, Status, Label) \ > do { \ > if (!(Expression)) { \ > gActiveConstraintHandler (__FILE__, __LINE__, Expression, Status); \ > goto Label; \ > } \ > } while (FALSE) > > > As a caller we have now have control: > EFI_STATUS Status; > CHAR16 Dst[2]; > > SetConstraintHandler (IgnoreConstraintHandler); > Status = StrCpyS (Dst, sizeof (Dst), L"Too Long"); > Print (L"Dst =%s (%r)\n", Dst, Status); > > SetConstraintHandler (AssertConstraintHandler); > StrCpyS (Dst, sizeof (Dst), L"Too Long"); > > Thanks, > > Andrew Fish > > PS Since I'm on a crazy idea roll another idea would be to add a > MdePkgCompatability.dsc.inc file that could be used to future proof > adding dependent libs to existing MdePkg libs. So a platform could > include this .DSC and that would give them the default library mapping > to keep code compiling. It will only work after other platforms start > including it, but after that it would give default mappings for > dependent libs. > > In our above example we could have added this and then existing builds > that included MdePkgCompatability.dsc.inc would keep compiling. > > [LibraryClasses] > > DebugConstraintLib|MdePkg/Library/DebugConstraintLib/DebugConstraintLib.inf > > > >> On Feb 15, 2020, at 11:38 AM, Michael D Kinney >> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> wrote: >> >> Andrew, >> I do not think of this as a globally scoped PCD.It can be set in >> global scope in DSC.But is can also be set to values based on module >> type or for specific modules.In the case of the safe string functions, >> I think there is a desire to disable the constraint asserts when >> building a UEFI App or UEFI Driver and implement those modules to >> handle the error return values.Enabling the constraint asserts for PEI >> Core, DXE Core, SMM/MM Core, PEIM, DXE, SMM/MM modules makes sense to >> find incorrect input to these functions from modules that can >> guarantee the inputs would never return an error and catch these as >> part of dev/debug/validation builds. >> I would not expect disabling on a module by module basis to be common. >> I think the rule for API implementations is to only use >> CONSTRAINT_ASSERT() for conditions that are also checked and return an >> error or fail with predicable behavior that allows the system to >> continue to function.ASSERT() is for conditions that the systemcan >> notcontinue. >> Best regards, >> Mike >> *From:*afish@apple.com <mailto:afish@apple.com><afish@apple.com >> <mailto:afish@apple.com>> >> *Sent:*Friday, February 14, 2020 10:27 PM >> *To:*vit9696 <vit9696@protonmail.com <mailto:vit9696@protonmail.com>> >> *Cc:*devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Kinney, >> Michael D <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>>; Gao, Liming >> <liming.gao@intel.com <mailto:liming.gao@intel.com>>; Gao, Zhichao >> <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>; Marvin Häuser >> <marvin.haeuser@outlook.com <mailto:marvin.haeuser@outlook.com>>; >> Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>> >> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to disable safe >> string constraint assertions >> Vitaly, >> Sorry after I sent the mail I realized it may come across as me >> asking you to do work and that was not my intent. >> I will point out though that a non backward compatible change to >> something as fundamental as the DebugLib is a very big deal. I've got >> a few different custom implementations that would break with this >> change as Mike proposed. Given breaking every one's debug lib is such >> a big deal maybe it is something that we should do as a long term plan >> vs. some incremental fix. So my intent was to start a conversation >> about what else we might want to change if we are going to break the >> world. The only think worse than breaking the world is breaking the >> world frequently. >> I'm also a little worried that we are taking things that are today >> locally scoped like SAFE_STRING_CONSTRAINT_CHECK and >> SAFE_PRINT_CONSTRAINT_CHECK and making them global constructs. I think >> the way others have dealt with things like this is to make them be >> DEBUG prints vs. ASSERTs. Also even something as simple as >> SAFE_STRING_CONSTRAINT_CHECK could be called from code that wants >> ASSERT and CONSTRAINT_ASSERT behavior. It is not clear to me that the >> low level code knows the right thing to do in a global sense even if >> there is a PCD. It almost seems like we should have wrappers for the >> Safe string functions that implement the behavior you want as a >> caller. I'm not sure about that, but it seems like it is worth talking >> about? >> Thanks, >> Andrew Fish >> >> >> On Feb 14, 2020, at 7:31 PM, vit9696 <vit9696@protonmail.com >> <mailto:vit9696@protonmail.com>> wrote: >> Hi Andrew, >> While your suggestions look interesting, I am afraid they are not >> particularly what we want to cover with this discussion at the moment. >> Making assertions go through DEBUG printing functions/macros is >> what we have to strongly disagree about. Assertions and debug >> prints are separate things configurable by separate PCDs. We >> should not mix them. Introducing constraint assertions is a >> logical step forward in understanding that different software >> works in different environments. >> >> * There are normal, or, as I call them, invariant >> assertions (e.g. preconditions), for places where the function >> cannot work unless the assertion is satisfied. This is where >> we ASSERT. >> * There are constraint assertions, which signalise that bad data >> came through the function, even though the function was called >> from a trusted source. This is where we call CONSTRAINT_ASSERT. >> >> The thing we need is to have the latter separable >> and configurable, because not every piece of software works in a >> trusted environment. Other than that, constraint assertions, when >> enabled, are not anyhow different from normal assertions in the >> sense of action taken. Assertions have configurable breakpoints >> and deadloops, and DEBUG prints go through a different route in >> DebugLib that may cause entirely different effects. For example, >> we halt execution upon printing to DEBUG_ERROR in our DebugLib >> even in release builds. >> >> =To make it clear, currently I plan to add the following interface: >> #define CONSTRAINT_ASSERT(Expression) \ >> do { \ >> if (DebugConstraintAssertEnabled ()) { \ >> if (!(Expression)) { \ >> _ASSERT (Expression); \ >> ANALYZER_UNREACHABLE (); \ >> } \ >> } \ >> } while (FALSE) >> with DebugConstraintAssertEnabled implemented as >> (BOOLEAN) ((PcdGet8(PcdDebugPropertyMask) & >> DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED | >> DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED) == >> DEBUG_PROPERTY_DEBUG_ASSERT_ENABLED) >> Your suggestion with require macros looks interesting indeed, but >> I believe it is in fact parallel to this discussion. The change we >> discuss introduces a new assertion primitive — constraint >> assertions, while REQUIRE macros are mostly about advanced syntax >> sugar and higher level assertion primitives on top of existing >> ones. Perhaps we can have this and make a good use of it, >> especially given that it brought some practical benefit in Apple, >> but I would rather discuss this later once constraint assertions >> are merged into EDK II tree. >> Best wishes, >> Vitaly >> On Sat, Feb 15, 2020 at 03:02, Andrew Fish <afish@apple.com >> <mailto:afish@apple.com>> wrote: >> >> >> >> On Feb 14, 2020, at 2:50 PM, Michael D Kinney >> <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>> wrote: >> Hi Vitaly, >> I agree that this proposal makes a lot of sense. We >> recently added a new assert type called STATIC_ASSERT() >> for assert conditions that can be tested at build time. >> A new assert type for checks that can be removed and the >> API still guarantees that it fails gracefully with a >> proper return code is a good idea. Given we have >> STATIC_ASSERT(), how about naming the new macro >> CONSTRAINT_ASSERT()? >> We also want the default to be enabled. The current use of >> bit 0x40 inPcdDebugPropertyMask is always clear, so we >> want the asserts enabled when 0x40 is clear. We can change >> the name of the define bit to >> DEBUG_PROPERTY_CONTRAINT_ASSERT_DISABLED so bit 0x40 needs >> to be set inPcdDebugPropertyMaskto disable these types of >> asserts. >> This approach does require all theDebugLibimplementations >> to be updated with the newDebugConstraintAssertDisabled() API. >> >> Mike, >> If you wanted to be backward compatible you could just >> use DebugAssertEnabled () but in place of _ASSERT() use >> _CONSTRAINT_ASSERT >> #define _ASSERT(Expression) DebugAssert (__FILE__, __LINE__, >> #Expression) >> #define _CONSTRAINT_ASSERT(Expression) DebugPrint >> (DEBUG_ERROR, "ASSERT %a(%d): %a\n",, __FILE__, __LINE__, >> #Expression) >> Not as elegant as the non backward compatible change, but I >> thought I'd throw it out there. >> There are some ancient Apple C ASSERT macros [AssertMacros.h] >> that also have the concept of require. Require includes an >> exception label (goto label). It is like a CONSTRAINT_ASSERT() >> but with the label. On release builds the DEBUG prints are >> skipped. >> So we could do something like: >> EFI_STATUS Status = EFI_INVALID_PARAMETER; >> REQUIRE(Arg1 != NULL, ErrorExit); >> REQUIRE(Arg2 != NULL, ErrorExit); >> REQUIRE(Arg3 != NULL, ErrorExit); >> ErrorExit: >> return Status; >> There is another form that allows an ACTION (a statement to >> execute. So you could have: >> EFI_STATUS Status; >> REQUIRE_ACTION(Arg1 != NULL, ErrorExit, Status = >> EFI_INVALID_PARAMETER); >> REQUIRE_ACTION(Arg2 != NULL, ErrorExit, Status = >> EFI_INVALID_PARAMETER); >> REQUIRE_ACTION(Arg3 != NULL, ErrorExit, Status = >> EFI_INVALID_PARAMETER); >> ErrorExit: >> return Status; >> If you use CONSTRAINT_ASSERT(); >> if (Arg1 == NULL || Arg2 == NULL || Arg3 == NULL) { >> CONSTRAINT_ASSERT (Arg1 != NULL); >> CONSTRAINT_ASSERT (Arg2 != NULL); >> CONSTRAINT_ASSERT (Arg3 != NULL); >> return EFI_INVALID_PARAMETER; >> } >> I'd note error processing args on entry is the simplest case. >> In a more complex case when cleanup is required the goto >> label is more useful. >> I guess we could argue for less typing and more symmetry and >> say use ASSERT, CONSTRAINT, and REQUIRE. I guess you could add >> an ASSERT_ACTION too. >> The AssertMacros.h versions also support _quiet (skip the >> print) and _string (add a string to the print) so you end up with: >> REQUIRE >> REQUIRE_STRING >> REQUIRE_QUIET >> REQUIRE_ACTION >> REQUIRE_ACTION_STRING >> REQUIRE_ACTION_QUIET >> We could also end up with >> CONSTRAINT >> CONSTRAINT_STRING >> CONSTRAINT_QUIET >> I think the main idea behind _QUIET is you can silence things >> that are too noisy, and you can easily make noise things show >> up by removing the _QUIET to debug. >> I'd thought I throw out the other forms for folks to think >> about. I'm probably biased as I used to looking at code and >> seeing things like require_action_string(Arg1 != NULL, >> ErrorExit, Status = EFI_INVALID_PARAMETER, "1st Arg1 check"); >> Thanks, >> Andrew Fish >> PS The old debug macros had 2 versions of CONSTRAINT check and >> verify. The check version was compiled out on a release build, >> the verify version always does the check and just skips the >> DEBUG print. >> >> Mike >> *From:*vit9696 <vit9696@protonmail.com >> <mailto:vit9696@protonmail.com>> >> *Sent:*Friday, February 14, 2020 9:38 AM >> *To:*Kinney, Michael D <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>> >> *Cc:*devel@edk2.groups.io <mailto:devel@edk2.groups.io>; >> Gao, Liming <liming.gao@intel.com >> <mailto:liming.gao@intel.com>>; Gao, Zhichao >> <zhichao.gao@intel.com <mailto:zhichao.gao@intel.com>>; >> Marvin Häuser <marvin.haeuser@outlook.com >> <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek >> <lersek@redhat.com <mailto:lersek@redhat.com>> >> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >> disable safe string constraint assertions >> Michael, >> Generalising the approach makes good sense to me, but we >> need to make an obvious distinguishable difference between: >> - precondition and invariant assertions (i.e. assertions, >> where function will NOT work if they are violated) >> - constraint asserts (i.e. assertions, which allow us to >> spot unintentional behaviour when parsing untrusted data, >> but which do not break function behaviour). >> As we want to use this outside of SafeString, I suggest >> the following: >> - Introduce DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED 0x40 >> bit for PcdDebugPropertyMask instead >> of PcdAssertOnSafeStringConstraints. >> - Introduce DebugAssertConstraintEnabled DebugLib function >> to check for DEBUG_PROPERTY_ASSERT_CONSTRAINT_ENABLED. >> - Introduce ASSERT_CONSTRAINT macro, that will assert only >> if DebugConstraintAssertEnabled returns true. >> - Change SafeString ASSERTS >> in SAFE_STRING_CONSTRAINT_CHECK to ASSERT_CONSTRAINTs. >> - Use ASSERT_CONSTRAINT in other places where necessary. >> >> I believe this way lines best with EDK II design. If there >> are no objections, I can submit the patch in the beginning >> of next week. >> >> Best wishes, >> Vitaly >> >> 14 ????. 2020 ?., ? 20:00, Kinney, Michael D >> <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>> ???????(?): >> Vitaly, >> I want to make sure a feature PCD can be used to >> disable ASSERT() behavior in more than just safe >> string functions inBaseLib. >> Can we consider changing the name and description >> ofPcdAssertOnSafeStringConstraintsto be more generic, >> so if we find other lib APIs, the name will make sense? >> Maybe something like:PcdEnableLibraryAssertChecks? >> Default is TRUE. Can set to FALSE in DSC file to >> disable ASSERT() checks. >> Thanks, >> Mike >> *From:*devel@edk2.groups.io >> <mailto:devel@edk2.groups.io><devel@edk2.groups.io >> <mailto:devel@edk2.groups.io>>*On Behalf Of*Vitaly >> Cheptsov via Groups.Io >> *Sent:*Friday, February 14, 2020 3:55 AM >> *To:*Kinney, Michael D <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>>; Gao, Liming >> <liming.gao@intel.com <mailto:liming.gao@intel.com>>; >> Gao, Zhichao <zhichao.gao@intel.com >> <mailto:zhichao.gao@intel.com>>;devel@edk2.groups.io >> <mailto:devel@edk2.groups.io> >> *Cc:*Marvin Häuser <marvin.haeuser@outlook.com >> <mailto:marvin.haeuser@outlook.com>>; Laszlo Ersek >> <lersek@redhat.com <mailto:lersek@redhat.com>> >> *Subject:*Re: [edk2-devel] [PATCH v3 0/1] Add PCD to >> disable safe string constraint assertions >> Replying as per Liming's request for this to be merged >> into edk2-stable202002. >> On Mon, Feb 10, 2020 at 14:12, vit9696 >> <vit9696@protonmail.com >> <mailto:vit9696@protonmail.com>> wrote: >> >> Hello, >> >> It has been quite some time since we submitted the >> patch with so far no negative response. As I >> mentioned previously, my team will strongly >> benefit from its landing in EDK II mainline. Since >> it does not add any regressions and can be viewed >> as a feature implementation for the rest of EDK II >> users, I request this to be merged upstream in >> edk2-stable202002. >> >> Best wishes, >> Vitaly >> >> > 27 ???. 2020 ?., ? 12:47, vit9696 >> <vit9696@protonmail.com >> <mailto:vit9696@protonmail.com>> ???????(?): >> > >> > >> > Hi Mike, >> > >> > Any progress with this? We would really benefit >> from this landing in the next stable release. >> > >> > Best, >> > Vitaly >> > >> >> 8 ???. 2020 ?., ? 19:35, Kinney, Michael D >> <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>> ???????(?): >> >> >> >> >> >> Hi Vitaly, >> >> >> >> Thanks for the additional background. I would like >> >> a couple extra day to review the PCD name and >> the places >> >> the PCD might potentially be used. >> >> >> >> If we find other APIs where ASSERT() behavior >> is only >> >> valuable during dev/debug to quickly identify >> misuse >> >> with trusted data and the API provides predicable >> >> return behavior when ASSERT() is disabled, then >> I would >> >> like to have a pattern we can potentially apply >> to all >> >> these APIs across all packages. >> >> >> >> Thanks, >> >> >> >> Mike >> >> >> >>> -----Original Message----- >> >>> From:devel@edk2.groups.io >> <mailto:devel@edk2.groups.io><devel@edk2.groups.io >> <mailto:devel@edk2.groups.io>> On >> >>> Behalf Of Vitaly Cheptsov via Groups.Io >> >>> Sent: Monday, January 6, 2020 10:44 AM >> >>> To: Kinney, Michael D >> <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>> >> >>> Cc:devel@edk2.groups.io >> <mailto:devel@edk2.groups.io> >> >>> Subject: Re: [edk2-devel] [PATCH v3 0/1] Add >> PCD to >> >>> disable safe string constraint assertions >> >>> >> >>> Hi Mike, >> >>> >> >>> Yes, the primary use case is for UEFI >> Applications. We >> >>> do not want to disable ASSERT’s completely, as >> >>> assertions that make sense, i.e. the ones >> signalising >> >>> about interface misuse, are helpful for debugging. >> >>> >> >>> I have already explained in the BZ that >> basically all >> >>> safe string constraint assertions make no >> sense for >> >>> handling untrusted data. We find this use case >> very >> >>> logical, as these functions behave properly with >> >>> assertions disabled and cover all these error >> >>> conditions by the return statuses. In such >> situation is >> >>> not useful for these functions to assert, as >> we end up >> >>> inefficiently reimplementing the logic. I >> would have >> >>> liked the approach of discussing the interfaces >> >>> individually, but I struggle to find any that >> makes >> >>> sense from this point of view. >> >>> >> >>> AsciiStrToGuid will ASSERT when the length of the >> >>> passed string is odd. Functions that cannot, ahem, >> >>> parse, for us are pretty much useless. >> >>> AsciiStrCatS will ASSERT when the appended >> string does >> >>> not fit the buffer. For us this logic makes this >> >>> function pretty much equivalent to deprecated >> and thus >> >>> unavailable AsciiStrCat, except it is also slower. >> >>> >> >>> My original suggestion was to remove the >> assertions >> >>> entirely, but several people here said that >> they use >> >>> them to verify usage errors when handling >> trusted data. >> >>> This makes good sense to me, so we suggest to >> support >> >>> both cases by introducing a PCD in this patch. >> >>> >> >>> Best wishes, >> >>> Vitaly >> >>> >> >>>> 6 ???. 2020 ?., ? 21:28, Kinney, Michael D >> >>> <michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>> ???????(?): >> >>>> >> >>>> >> >>>> Hi Vitaly, >> >>>> >> >>>> Is the use case for UEFI Applications? >> >>>> >> >>>> There is a different mechanism to disable all >> >>> ASSERT() >> >>>> statements within a UEFI Application. >> >>>> >> >>>> If a component is consuming data from an >> untrusted >> >>> source, >> >>>> then that component is required to verify the >> >>> untrusted >> >>>> data before passing it to a function that clearly >> >>> documents >> >>>> is input requirements. If this approach is >> followed, >> >>> then >> >>>> the BaseLib functions can be used "as is" as >> long as >> >>> the >> >>>> ASSERT() conditions are verified before calling. >> >>>> >> >>>> If there are some APIs that currently >> document their >> >>> ASSERT() >> >>>> behavior and we think that ASSERT() behavior is >> >>> incorrect and >> >>>> should be handled by an existing error return >> value, >> >>> then we >> >>>> should discuss each of those APIs individually. >> >>>> >> >>>> Mike >> >>>> >> >>>> >> >>>>> -----Original Message----- >> >>>>> From:devel@edk2.groups.io >> <mailto:devel@edk2.groups.io><devel@edk2.groups.io >> <mailto:devel@edk2.groups.io>> On >> >>>>> Behalf Of Vitaly Cheptsov via Groups.Io >> >>>>> Sent: Friday, January 3, 2020 9:13 AM >> >>>>> To:devel@edk2.groups.io >> <mailto:devel@edk2.groups.io> >> >>>>> Subject: [edk2-devel] [PATCH v3 0/1] Add PCD to >> >>> disable >> >>>>> safe string constraint assertions >> >>>>> >> >>>>> REF: >> >>>>>https://bugzilla.tianocore.org/show_bug.cgi?id=2054 >> >>>>> >> >>>>> Requesting for merge in edk2-stable202002. >> >>>>> >> >>>>> Changes since V1: >> >>>>> - Enable assertions by default to preserve the >> >>> original >> >>>>> behaviour >> >>>>> - Fix bugzilla reference link >> >>>>> - Update documentation in BaseLib.h >> >>>>> >> >>>>> Vitaly Cheptsov (1): >> >>>>> MdePkg: Add PCD to disable safe string >> constraint >> >>>>> assertions >> >>>>> >> >>>>> MdePkg/MdePkg.dec | 6 ++ >> >>>>> MdePkg/Library/BaseLib/BaseLib.inf | 11 +-- >> >>>>> MdePkg/Include/Library/BaseLib.h | 74 >> >>>>> +++++++++++++------- >> >>>>> MdePkg/Library/BaseLib/SafeString.c | 4 +- >> >>>>> MdePkg/MdePkg.uni | 6 ++ >> >>>>> 5 files changed, 71 insertions(+), 30 >> deletions(-) >> >>>>> >> >>>>> -- >> >>>>> 2.21.0 (Apple Git-122.2) >> >>>>> >> >>>>> >> >>>>> -=-=-=-=-=-= >> >>>>>Groups.io <http://groups.io/>Links: You >> receive all messages sent to >> >>> this >> >>>>> group. >> >>>>> >> >>>>> View/Reply Online (#52837): >> >>>>>https://edk2.groups.io/g/devel/message/52837 >> >>>>> Mute This Topic: >> >>>https://groups.io/mt/69401948/1643496 >> >>>>> Group Owner:devel+owner@edk2.groups.io >> <mailto:devel+owner@edk2.groups.io> >> >>>>> Unsubscribe:https://edk2.groups.io/g/devel/unsub >> >>>>> [michael.d.kinney@intel.com >> <mailto:michael.d.kinney@intel.com>] >> >>>>> -=-=-=-=-=-= >> >>>> >> >>> >> >>> >> >>> >> >> >> > >> > > [-- Attachment #2: Type: text/html, Size: 85448 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
end of thread, other threads:[~2020-03-04 0:19 UTC | newest]
Thread overview: 28+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-01-03 17:12 [PATCH v3 0/1] Add PCD to disable safe string constraint assertions Vitaly Cheptsov
2020-01-03 17:12 ` [PATCH v3 1/1] MdePkg: " Vitaly Cheptsov
2020-01-06 18:28 ` [edk2-devel] [PATCH v3 0/1] " Michael D Kinney
2020-01-06 18:43 ` Vitaly Cheptsov
2020-01-06 22:54 ` Sean
2020-01-08 16:35 ` Michael D Kinney
2020-01-27 9:47 ` Vitaly Cheptsov
2020-02-10 11:12 ` Vitaly Cheptsov
2020-02-14 11:54 ` Vitaly Cheptsov
2020-02-14 17:00 ` Michael D Kinney
2020-02-14 17:37 ` Vitaly Cheptsov
2020-02-14 22:50 ` Michael D Kinney
2020-02-14 23:04 ` Vitaly Cheptsov
2020-02-15 0:02 ` Andrew Fish
2020-02-15 3:31 ` Vitaly Cheptsov
2020-02-15 6:26 ` Andrew Fish
2020-02-15 11:53 ` Vitaly Cheptsov
2020-02-15 12:02 ` Vitaly Cheptsov
2020-02-15 19:38 ` Michael D Kinney
2020-02-16 21:25 ` Andrew Fish
2020-02-17 6:55 ` Vitaly Cheptsov
2020-02-17 8:26 ` Marvin Häuser
2020-02-19 23:55 ` Andrew Fish
2020-02-20 10:18 ` Marvin Häuser
2020-03-03 19:38 ` Marvin Häuser
2020-03-04 0:19 ` Liming Gao
2020-03-03 20:27 ` Andrew Fish
[not found] ` <15F4232304E080CF.5373@groups.io>
2020-02-17 9:36 ` Marvin Häuser
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox