From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mx.groups.io with SMTP id smtpd.web11.68415.1584367230452187058 for ; Mon, 16 Mar 2020 07:00:30 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.136, mailfrom: ray.ni@intel.com) IronPort-SDR: by6g8vY+vq4qF2Ek1dLuCqbNSrqmdrcTaGi91wLOKcwqNYukwfQ3iJSflHd5t4kTopOdxUpgVw HJv9JLFonzeA== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2020 07:00:29 -0700 IronPort-SDR: uOMCekBxG0NW7lgjl7xBxKDUnZGD62OqRLRIqqcR7kzzEYQNnWpDOzKwZakIQWdUreOk7IEO1y vWfQ1LnkQfog== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,560,1574150400"; d="scan'208";a="417148371" Received: from fmsmsx107.amr.corp.intel.com ([10.18.124.205]) by orsmga005.jf.intel.com with ESMTP; 16 Mar 2020 07:00:29 -0700 Received: from fmsmsx153.amr.corp.intel.com (10.18.125.6) by fmsmsx107.amr.corp.intel.com (10.18.124.205) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 16 Mar 2020 07:00:29 -0700 Received: from shsmsx153.ccr.corp.intel.com (10.239.6.53) by FMSMSX153.amr.corp.intel.com (10.18.125.6) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 16 Mar 2020 07:00:28 -0700 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.206]) by SHSMSX153.ccr.corp.intel.com ([169.254.12.96]) with mapi id 14.03.0439.000; Mon, 16 Mar 2020 22:00:26 +0800 From: "Ni, Ray" To: "Javeed, Ashraf" , "devel@edk2.groups.io" CC: "Wang, Jian J" , "Wu, Hao A" Subject: Re: [edk2-devel] [edk2-staging/UEFI_PCI_ENHANCE-2 PATCH 05/12] PciBusDxe: Setup sub-phases for PCI feature enumeration Thread-Topic: [edk2-devel] [edk2-staging/UEFI_PCI_ENHANCE-2 PATCH 05/12] PciBusDxe: Setup sub-phases for PCI feature enumeration Thread-Index: AQHVmdJK3T9Y1rxzHkawV8SqYkiDX6e92zmAgAFQkwCAAfiR8IB5l2IAgBB2hwCAAMscYA== Date: Mon, 16 Mar 2020 14:00:25 +0000 Message-ID: <734D49CCEBEEF84792F5B80ED585239D5C49D05C@SHSMSX104.ccr.corp.intel.com> References: <20191101150952.3340-1-ashraf.javeed@intel.com> <15D3127B934F51D3.12315@groups.io> <95C5C2B113DE604FB208120C742E9824579171C4@BGSMSX101.gar.corp.intel.com> <734D49CCEBEEF84792F5B80ED585239D5C3A05B7@SHSMSX104.ccr.corp.intel.com> <95C5C2B113DE604FB208120C742E98245797C292@BGSMSX101.gar.corp.intel.com> <15E1AFB3EABD031C.30484@groups.io> <734D49CCEBEEF84792F5B80ED585239D5C47E0CF@SHSMSX104.ccr.corp.intel.com> <95C5C2B113DE604FB208120C742E9824579C6B6A@BGSMSX101.gar.corp.intel.com> In-Reply-To: <95C5C2B113DE604FB208120C742E9824579C6B6A@BGSMSX101.gar.corp.intel.com> Accept-Language: en-US, zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.2.0.6 dlp-reaction: no-action x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Return-Path: ray.ni@intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Ashraf, Thanks for taking time to review my code! Comments embedded in below. > -----Original Message----- > From: Javeed, Ashraf > Sent: Monday, March 16, 2020 5:34 PM > To: Ni, Ray ; devel@edk2.groups.io > Cc: Wang, Jian J ; Wu, Hao A > Subject: RE: [edk2-devel] [edk2-staging/UEFI_PCI_ENHANCE-2 PATCH 05/12] = PciBusDxe: Setup sub-phases for PCI feature > enumeration >=20 > Ray, > First of all, thank you for taking time to review and on optimizing the = PCIe feature support in the PciBusDxe. >=20 > I shall not discuss everything in detail here, just want to bring two ma= jor points:- >=20 > (1) commit:- MdePkg: New PCI Express Platform/Override Protocols > https://github.com/niruiyu/edk2/commit/52524e9654704a7f6a30ca446f215b81f= e8f0984 > PCI Express Protocol not as per the ECR draft revision 0.8, the changes = made has to be reviewed with updated ECR > version... With the code first process, we could firstly finalize the protocol interf= aces in code then update the ECR. Code first doesn't mean implementation detail has high priority than proto= col interfaces. Good quality of protocol interfaces is still desired. The q= uality of the protocol header file is always in high priority. > the global auto option introduces EFI_PCI_EXPRESS_DEVICE_POLICY_AUTO; is= used for MPS, MRRS, RO, NS, AtomicOp, LTR, > results in no EFI encodings required for these > features, as it can be used with actual values from the PciXX.h definiti= ons. This do result in lot of code savings. I take it as an agree of using the global AUTO. Thanks for that. >=20 > (2) commit:- MdeModulePkg/PciBus: Add the framework to init PCIE feature= s > https://github.com/niruiyu/edk2/commit/9fb9a3dcef06de98a76825e2fc0716744= 6ee6fd9 > The context handling of the PCIe feature MPS has fundamental issue as it= is set to be common for all the nodes of the > parent Root Bridge instance. The context has to be separate to each fir= st level nodes of a root bridge instance. For > example, a root bridge has 1 RCiEP and two Root ports, than there has to= be 3 separate context for each. Since Root port > can have its Endpoint device, or it can have a PCIe switch with 1 upstre= am and 2 or more downstream ports and to each > downstream port an Endpoint device can be connected; the context created= for Root Port of an bridge is used for all its > child hierarchy nodes; thus PCIe feature like Max_Payload_Size value can= be specific to each RCiEP, and each Root ports of > the Root Bridge handle. How do you propose to maintain separate context = for just first level nodes of the Root Bridge > handle? Supposing a root bridge contains 1 RCiEP and two root ports, the implement= ation ensures there will be 3 separate contexts for each. As you can see, e= very time the code starts enumeration from a RCiEP or a root port, a new Co= ntext[] is setup. Context[i] is associated with the feature _i_. >=20 > The EnumeratePcieDevices() is recursively calls itself for every PCI nod= e with a level advanced by 1. Thus, the usage of level > N and N+1 as parent and its child is wrong to me, when you have a PCIe s= witch below the Root Port, and N & N+1 will not > be direct relation between the upstream port and its second downstream p= ort...any reason why you have not used the > PCI_IO_DEVICE.Parent pointer to form relation between the parent and the= child? The level N and N+1 is needed for AtomicOp or Ltr feature (which I need to= go back to check the code). I want to avoid the individual feature scan/program routine checks device'= s parent/children. I understand that in PCIE spec, a switch consists of an upstream port and = several downstream ports. It means a switch populates a bridge which connects to the upstream and se= veral bridges under the upstream bridge which connect to the downstream. But from software perspective, I don't think we should care about that. Maybe I am wrong. Can you please give a real example that treating in my w= ay would cause some issues? >=20 > I liked the Pre & Post order flag that is used to processing parent-to-c= hild vs. child-to-parent nodes.=20 Actually I didn't have this flag in the first version of my code change. B= ut later I found it's needed for some features like LTR which needs to be p= rogrammed from rootbridge to endpoint according to spec. > You are calling the first > and last features as fakes, when you actually parse all the nodes for th= e GetDevicePolicy() and NotifyDeviceState(); to me it > is actually an un-named phase where all the nodes are queried or informe= d to the platform; that seems fine to save the > NULL on the mPcieFeatures[] table. Yes I put GetDevicePolicy() and NotifyDeviceState() in the feature array i= n my first version of code. And later I called the two separately for bette= r code readability. But it was my fault that I didn't update the commit mes= sage. It caused confusion to you. >=20 > Thanks > Ashraf >=20 > > -----Original Message----- > > From: Ni, Ray > > Sent: Thursday, March 5, 2020 7:43 PM > > To: devel@edk2.groups.io; Ni, Ray ; Javeed, Ashraf > > > > Cc: Wang, Jian J ; Wu, Hao A > > > > Subject: RE: [edk2-devel] [edk2-staging/UEFI_PCI_ENHANCE-2 PATCH > > 05/12] PciBusDxe: Setup sub-phases for PCI feature enumeration > > > > Ashraf, > > I think it might be better to describe my review comments with code > > implementation. > > Can you please check this branch where I did some modification based o= n > > your code? > > https://github.com/niruiyu/edk2/tree/pci/pcie2 > > > > Let's firstly align on the feature initialization framework implementa= tion. > > To be specific, this commit: > > MdeModulePkg/PciBus: Add the framework to init PCIE features > > https://github.com/niruiyu/edk2/commit/9fb9a3dcef06de98a76825e2fc0 > > 7167446ee6fd9 > > > > Thanks, > > Ray > > > > > -----Original Message----- > > > From: devel@edk2.groups.io On Behalf Of Ni, > > Ray > > > Sent: Thursday, December 19, 2019 1:49 PM > > > To: Javeed, Ashraf ; devel@edk2.groups.io > > > Cc: Wang, Jian J ; Wu, Hao A > > > > > > Subject: Re: [edk2-devel] [edk2-staging/UEFI_PCI_ENHANCE-2 PATCH > > > 05/12] > > > PciBusDxe: Setup sub-phases for PCI feature enumeration > > > > > > After I reviewed the patch of enabling MaxPayloadSize, MaxReadReqSiz= e > > > and more PCIE features, I can now understand the phases more than > > > earlier. > > > > > > Your patch proposed five phases: > > > // > > > // initial phase in configuring the other PCI features to record t= he > > primary > > > // root ports > > > // > > > PciFeatureRootBridgeScan, > > > // > > > // get the PCI device-specific platform policies and align with de= vice > > capabilities > > > // > > > PciFeatureGetDevicePolicy, > > > // > > > // align all PCI nodes in the PCI heirarchical tree > > > // > > > PciFeatureSetupPhase, > > > // > > > // finally override to complete configuration of the PCI feature > > > // > > > PciFeatureConfigurationPhase, > > > // > > > // PCI feature configuration complete > > > // > > > PciFeatureConfigurationComplete > > > > > > > > > I have several comments to the five phases. > > > 1. Scan phase is not do the scanning but creates a list to hold all > > > root ports under the root bridge. > > > 2. Root ports collection is not required by all of the features, onl= y > > > by MPS and MRRS. > > > But the collection always happens even when platform doesn't require > > > PciBus to initialize MPS or MRRS. > > > 3. GetDevicePolicy phase is not just call the GetDevicePolicy for ea= ch > > > device. It also reads the PCIE configuration space to get the device= 's > > > feature related capabilities, for some of the features. > > > > > > With that, I propose to define 4 phases: > > > 1. Initialize phase > > > This phase is similar to your Scan phase. > > > For some features, this phase does nothing. > > > For MPS and MRRS, this phase creates a list holding all root ports. > > > > > > 2. Scan phase > > > This phase is similar to your GetDevicePolicy phase. > > > For some features, this phase needs nothing do to. > > > For MPS and MRRS, this phase scan all devices and get the aligned > > > value of MPS or MRRS. > > > > > > 3. Program phase or Configuration phase This phase is similar to you= r > > > Configuration phase. > > > The Setup phase can be merged to this phase. > > > > > > 4. Finalize phase. > > > This phase is similar to your ConfigurationComplete phase. > > > This phase frees the resources occupied/allocated in Initialize phas= e. > > > For some of the features, this phase may do nothing. > > > > > > Each feature needs to provide function pointers for each phase and > > > NULL means the feature doesn't need to do anything in the specific > > > phase. > > > With that, we can define a structure: > > > Typedef struct { > > > BOOLEAN Enable; > > > PCIE_FEATURE_INITILAIZE Initialize; > > > PCIE_FEATURE_SCAN Scan; > > > PCIE_FEATURE_PROGRAM Program; > > > PCIE_FEATURE_FINALIZE Finalize; > > > } PCIE_FEATURE_ENTRY; > > > > > > With that, we can define a module level global variable: > > > PCIE_FEATURE_ENTRY mPcieFeatures[] =3D { > > > { TRUE, MaxPayloadInitialize, MaxPayloadScan, MaxPayloadProgram, > > > MaxPayloadFinalize}, > > > { TRUE, MaxReadRequestInitialize, MaxReadRequestScan, > > > MaxReadRequestProgram, MaxReadRequestFinalize}, > > > { TRUE, NULL, NULL, RelaxOrderProgram, NULL}, > > > { TRUE, NULL, CompletionTimeoutScan, CompletionTimeoutProgram, > > NULL }, > > > ... > > > }; > > > > > > PCIE_FEATURE_ENTRY.Enable can be set to FALSE according to the > > > platform policy. > > > > > > The enable of PCIE features can be written as a feature agnostic for= -loop. > > > This can make the new feature enabling code easy to add and review. > > > > > > > > > > -----Original Message----- > > > > From: Javeed, Ashraf > > > > Sent: Wednesday, December 18, 2019 3:14 PM > > > > To: Ni, Ray ; devel@edk2.groups.io > > > > Cc: Wang, Jian J ; Wu, Hao A > > > > > > > > Subject: RE: [edk2-devel] [edk2-staging/UEFI_PCI_ENHANCE-2 PATCH > > > > 05/12] > > > > PciBusDxe: Setup sub-phases for PCI feature enumeration > > > > > > > > Thanks for the review, Ray! > > > > My response in line > > > > > > > > > -----Original Message----- > > > > > From: Ni, Ray > > > > > Sent: Tuesday, December 17, 2019 5:26 PM > > > > > To: Javeed, Ashraf ; devel@edk2.groups.= io > > > > > Cc: Wang, Jian J ; Wu, Hao A > > > > > > > > > > Subject: RE: [edk2-devel] [edk2-staging/UEFI_PCI_ENHANCE-2 PATCH > > > > 05/12] > > > > > PciBusDxe: Setup sub-phases for PCI feature enumeration > > > > > > > > > > Please check comments below. > > > > > I may have more comments regarding to the four phases after I > > > > > finish > > > > review of > > > > > further patches. > > > > > > > > > > Besides the comments below, I have a general comments to the deb= ug > > > > > message: can you please review the existing debug message in the > > > > > PciBus > > > > driver > > > > > and make sure your newly added debug message aligns to existing > > style. > > > > And try > > > > > to use less lines of debug messages with still enough debug > > information. > > > > Ok, will look into that. > > > > > > > > > > > > > > > > +PRIMARY_ROOT_PORT_NODE > > *mPrimaryRootPortList; > > > > > > > + > > > > > > > +/** > > > > > > > + A global pointer to > > > > PCI_FEATURE_CONFIGURATION_COMPLETION_LIST, > > > > > > > which > > > > > > > +stores all > > > > > > > + the PCI Root Bridge instances that are enumerated for the > > > > > > > +other PCI features, > > > > > > > + like MaxPayloadSize & MaxReadReqSize; during the the > > > > > > > +Start() interface of the > > > > > > > + driver binding protocol. The records pointed by this > > > > > > > +pointer would be destroyed > > > > > > > + when the DXE core invokes the Stop() interface. > > > > > > > +**/ > > > > > > > +PCI_FEATURE_CONFIGURATION_COMPLETION_LIST > > > > > > > *mPciFeaturesConfigurationCompletionList =3D NULL; > > > > > > > > > > 1. Please follow existing linked list usage style. The first nod= e > > > > > in the list is an empty header node. > > > > > > > > > > LIST_ENTRY mPrimaryRootPortList; > > > > > LIST_ENTRY mPciFeaturesConfigurationCompletionList; > > > > > > > > > Ok, will make the change when I incorporate the ECR 0.75 or greate= r > > version. > > > > > > > > > > > +BOOLEAN > > > > > > > +CheckPciFeatureConfigurationRecordExist ( > > > > > > > + IN PCI_IO_DEVICE *RootBridge= , > > > > > > > + OUT PCI_FEATURE_CONFIGURATION_COMPLETION_LIST > > > > > > > +**PciFeatureConfigRecord > > > > > > > + ) > > > > > > > > > > 2. Is this function to check whether the PCIE features under a > > > > > root bridge is already initialized? > > > > > Can you use the existing variable gFullEnumeration? > > > > > The variable is set to TRUE when the enumeration is done to a ho= st > > > > > bridge > > > > in the > > > > > first time. > > > > > By using gFullEnumeration, the entire function is not needed. > > > > > > > > > Ok, will look into this. > > > > > > > > > > > +EFI_STATUS AddRootBridgeInPciFeaturesConfigCompletionList ( > > > > > > > + IN PCI_IO_DEVICE *RootBridge, > > > > > > > + IN BOOLEAN ReEnumerationRequired > > > > > > > + ) > > > > > > > > > > 3. Same question as #2. I think by using gFullEnumeration, this > > > > > function is > > > > not > > > > > needed. > > > > > > > > > OK > > > > > > > > > > > > > > > > +BOOLEAN > > > > > > > +IsPciRootPortEmpty ( > > > > > > > + IN PCI_IO_DEVICE *PciDevice > > > > > > > + ) > > > > > > > > > > 4. Please use IsListEmpty() directly from callers and remove thi= s > > function. > > > > > > > > > Will consider this. > > > > > > > > > > > +**/ > > > > > > > +EFI_STATUS > > > > > > > +EnumerateOtherPciFeatures ( > > > > > > > > > > 5. Can it be "EnumeratePcieFeatures"? > > > > > > > > > Yes, with the change to ECR 0.75, this routine name shall be chang= ed. > > > > > > > > > > > + IN PCI_IO_DEVICE *RootBridge > > > > > > > + ) > > > > > > > +{ > > > > > > > + EFI_STATUS Status; > > > > > > > + UINTN OtherPciFeatureConfigPhase; > > > > > > > + > > > > > > > + // > > > > > > > + // check on PCI features configuration is complete and > > > > > > > + re-enumeration is required // if > > > > > > > + (!CheckPciFeaturesConfigurationRequired > > > > > > > + (RootBridge)) { > > > > > > > + return EFI_ALREADY_STARTED; } > > > > > > > + > > > > > > > + CHAR16 *Str; > > > > > > > + Str =3D ConvertDevicePathToText ( > > > > > > > + DevicePathFromHandle (RootBridge->Handle), > > > > > > > + FALSE, > > > > > > > + FALSE > > > > > > > + ); > > > > > > > + DEBUG ((DEBUG_INFO, "Enumerating PCI features for Root > > > > > > > + Bridge %s\n", Str !=3D NULL ? Str : L"")); > > > > > > > > > > 6. Please use DEBUG_CODE macro to include > > > > > ConvertDevicePathToText() > > > > and > > > > > DEBUG(). > > > > > Please remember to call FreePool(). > > > > > > > > > Ok, will can under DEBUG_CODE, and free pool is called in the end > > > > > > > > > > > + > > > > > > > + for ( OtherPciFeatureConfigPhase =3D PciFeatureRootBridge= Scan > > > > > > > + ; OtherPciFeatureConfigPhase <=3D > > > > PciFeatureConfigurationComplete > > > > > > > + ; OtherPciFeatureConfigPhase++ > > > > > > > + ) { > > > > > > > + switch (OtherPciFeatureConfigPhase){ > > > > > > > + case PciFeatureRootBridgeScan: > > > > > > > + SetupPciFeaturesConfigurationDefaults (); > > > > > > > + // > > > > > > > + //first scan the entire root bridge heirarchy for t= he > > > > > > > + primary PCI root > > > > > > ports > > > > > > > + // > > > > > > > + RecordPciRootPortBridges (RootBridge); > > > > > > > > > > 7. How about "RecordPciRootPorts (RootBridge)"? The "Bridges" > > > > > suffix is a > > > > bit > > > > > confusing. > > > > > > > > > Fine, will change. > > > > > > > > > > > + case PciFeatureGetDevicePolicy: > > > > > > > + case PciFeatureSetupPhase: > > > > > > > > > > 8. In SetupPciFeatures(), why do you need to call DeviceExist()? > > > > > Did you see any case that a device is detected in the beginning = of > > > > > PciBus > > > > scan > > > > > but is hidden when calling SetupPciFeatures()? > > > > > > > > > Yes, that is the case; device detected during the beginning of > > > > PciBus scan appears to be hidden by the platform drivers, since > > > > numerous legacy callbacks are initiated at different phase of PCI > > > > enumeration to the PCI Host Bridge, and PciPlatform drivers. > > > > This can be avoided if the PciBus driver is enhanced to check for > > > > PCI device existence before the publication of the PCI IO Protocol= , > > > > and removal of the PCI_IO_DEVICE instance from the linked list. > > > > > > > > > 9. In GetPciFeaturesConfigurationTable() when checking whether a > > > > > PCI > > > > device > > > > > belongs to a root port, we can use below simpler logic: > > > > > SizeOfPciDevicePath =3D GetDevicePathSize (PciDevicePath); > > > > > SizeOfRootPortDevicePath =3D GetDevicePathSize (RootPortPath); > > > > > if ((SizeOfRootPortDevicePath < SizeOfPciDevicePath) && > > > > > CompareMem (PciDevicePath, RootPortPath, > > > > SizeOfRootPortDevicePath - > > > > > END_DEVICE_PATH_LENGTH) =3D=3D 0)) { > > > > > // PCI device belongs to the root port. > > > > > } > > > > > > > > > Ok. > > > > > > > > > > > + Status =3D ProgramPciFeatures (RootBridge); > > > > > 10. ProgramPcieFeatures()? > > > > > > > > > OK > > > > > > > > > > > + > > > > > > > + if (Str !=3D NULL) { > > > > > > > + FreePool (Str); > > > > > > > + } > > > > > > > > > > 11. OK the Str is freed here because Str is needed for other deb= ug > > > > messages > > > > > inside the function. > > > > > > > > > Yes > > > > > > > > > > > + // > > > > > > > + // mark this root bridge as PCI features configuration > > > > > > > +complete, and no new > > > > > > > + // enumeration is required > > > > > > > + // > > > > > > > + AddRootBridgeInPciFeaturesConfigCompletionList (RootBridg= e, > > > > > > > +FALSE); > > > > > > > > > > 12. Not needed. > > > > > > > > > ok, after incorporating the logic of gFullEnumeration it won't be > > > > required > > > > > > > > > > > +_PRIMARY_ROOT_PORT_NODE { > > > > > > > > > > > > + // > > > > > > > + // Signature header > > > > > > > + // > > > > > > > + UINT32 Signature; > > > > > > > + // > > > > > > > + // linked list pointers to next node > > > > > > > + // > > > > > > > + LIST_ENTRY NeighborRootPor= t; > > > > > > > + // > > > > > > > + // pointer to PCI_IO_DEVICE of the primary PCI Controller > > > > > > > +device > > > > > > > + // > > > > > > > + EFI_DEVICE_PATH_PROTOCOL *RootPortDevice= Path; > > > > > > > + // > > > > > > > + // pointer to the corresponding PCI feature configuration > > > > > > > +Table node > > > > > > > + // all the child PCI devices of the controller are aligne= d > > > > > > > +based on this table > > > > > > > + // > > > > > > > + OTHER_PCI_FEATURES_CONFIGURATION_TABLE > > > > > > > *OtherPciFeaturesConfigurationTable; > > > > > > > +}; > > > > > > > > > > 13. Can you add the OTHER_PCI_FEATURES_CONFIGURATION_TABLE > > field > > > > to > > > > > PCI_IO_DEVICE structure? > > > > > So this structure PRIMARY_ROOT_PORT_NODE is not needed. > > > > > > > > > I think it is better to maintain separately as this configuration > > > > table is confined to a group of PCI devices and for the RCiEP it i= s > > > > not applicable hence not required. Moreover, I am maintaining a > > > > variable for each PCIe feature in the PCI_IO_DEVICE; perhaps I can > > consider having just pointer of it.... > > > > > > > > > > > +struct _PCI_FEATURE_CONFIGURATION_COMPLETION_LIST { > > > > > 14. This structure is not needed if using gFullEnumeration. > > > > Yes. > > > > > >=20