[RFC] Proposal to split Pkgs

[RFC] Proposal to split Pkgs

[Ni, Ray]

Hello,
I'd like to propose to split today's BIG packages in following ways:

==============Overview =================

1. Separate Industry standard definitions from UEFI and PI interfaces.
2. Separate UEFI and PI interfaces from implementations.
    a. Separate UEFI and PI interfaces to different packages
    b. Separate PI PEI, DXE and MM phase interfaces to different packages
3. Separate different features into feature packages.
    Feature interface may be in the feature package to provide minimal
    common interface packages.

The POC code is in https://github.com/jyao1/edk2/tree/ReOrg.
It basically split the EDKII common code to three directories:
Core/, Device/, and Feature/.
The code is in very early POC phase and only code in Core/ directory
can pass the build.
I would like to gather feedbacks through this RFC to see whether
this splitting direction makes sense.
Is there any other better ways of splitting?
Or perhaps do not split in such a small granularity?
Or perhaps Mike's work to move lib-c content to edk2-libc repo,
to move real platform code to edk2-platform repo is enough for
now?

==============More explanations =================

####There are 9 packages inside Core/ directory:
1. BasePkg
Contains industry standard definitions (exclude UEFI and PI) and base
libraries that non-UEFI and non-PI development can depend on.
UEFI or PI development can also depend on this package.
2. UefiPkg
Contains UEFI interfaces and libraries that UEFI driver-model driver
development can depend on.
3. PiPeiPkg
Contains PI interfaces and libraries for PEI phase that PEI module
development can depend on.
4. PiDxePkg
Contains PI interfaces and libraries for DXE phase that DXE module
development can depend on.
5. PiMmPkg
Contains PI interfaces and libraries for MM phase that MM/SMM
module development can depend on.
6. MdeModulePkg (TianoPkg? Name is still TBD)
Contains Tiano implementation specific interfaces and libraries.
Developing modules for pure UEFI or PI should not depend on this package.
7. PeiFoundationPkg
Contains the PEI foundation modules (PeiCore and DxeIpl) and supported
libraries.
8. DxeFoundationPkg
Contains the DXE foundation modules (DxeCore and RuntimeDxe) and
supported libraries.
9. SmmFoundationPkg
Contains the SMM foundation modules (SmmCore, SmmIpl and
SmmCommunicationBuffer) and supported libraries.

These packages are positioned in different layers. The package in higher
layer depends on all packages that are in lower layers.
Layer 0: BasePkg.
Layer 1: UefiPkg.
Layer 2: PiPeiPkg 
Layer 3: PiDxePkg
Layer 4: PiMmPkg
Layer 5: MdeModulePkg (TianoPkg?)

####All other modules are put to small packages under Device/ or Feature/.

============== Benefit of this proposal =================

This helps to reduce the size of each package, especially the very BIG
MdeModulePkg which contains almost all edk2 modules (except
CPU, network, etc). So platform can use git sparse checkout feature
to only clone the needed code still in package granularity.
This also helps to separate the code maintenance to more expert
developers. MdeModulePkg is just too huge to be maintained by 2 or 3
developers.

Thanks,
Ray

Re: [RFC] Proposal to split Pkgs

[Laszlo Ersek]

Hi Ray,

On 01/29/19 06:59, Ni, Ray wrote:
> Hello,
> I'd like to propose to split today's BIG packages in following ways:
> 
> ==============Overview =================
> 
> 1. Separate Industry standard definitions from UEFI and PI interfaces.
> 2. Separate UEFI and PI interfaces from implementations.
>     a. Separate UEFI and PI interfaces to different packages
>     b. Separate PI PEI, DXE and MM phase interfaces to different packages
> 3. Separate different features into feature packages.
>     Feature interface may be in the feature package to provide minimal
>     common interface packages.
> 
> The POC code is in https://github.com/jyao1/edk2/tree/ReOrg.
> It basically split the EDKII common code to three directories:
> Core/, Device/, and Feature/.
> The code is in very early POC phase and only code in Core/ directory
> can pass the build.
> I would like to gather feedbacks through this RFC to see whether
> this splitting direction makes sense.
> Is there any other better ways of splitting?
> Or perhaps do not split in such a small granularity?
> Or perhaps Mike's work to move lib-c content to edk2-libc repo,
> to move real platform code to edk2-platform repo is enough for
> now?
> 
> ==============More explanations =================
> 
> ####There are 9 packages inside Core/ directory:
> 1. BasePkg
> Contains industry standard definitions (exclude UEFI and PI) and base
> libraries that non-UEFI and non-PI development can depend on.
> UEFI or PI development can also depend on this package.
> 2. UefiPkg
> Contains UEFI interfaces and libraries that UEFI driver-model driver
> development can depend on.
> 3. PiPeiPkg
> Contains PI interfaces and libraries for PEI phase that PEI module
> development can depend on.
> 4. PiDxePkg
> Contains PI interfaces and libraries for DXE phase that DXE module
> development can depend on.
> 5. PiMmPkg
> Contains PI interfaces and libraries for MM phase that MM/SMM
> module development can depend on.
> 6. MdeModulePkg (TianoPkg? Name is still TBD)
> Contains Tiano implementation specific interfaces and libraries.
> Developing modules for pure UEFI or PI should not depend on this package.
> 7. PeiFoundationPkg
> Contains the PEI foundation modules (PeiCore and DxeIpl) and supported
> libraries.
> 8. DxeFoundationPkg
> Contains the DXE foundation modules (DxeCore and RuntimeDxe) and
> supported libraries.
> 9. SmmFoundationPkg
> Contains the SMM foundation modules (SmmCore, SmmIpl and
> SmmCommunicationBuffer) and supported libraries.
> 
> These packages are positioned in different layers. The package in higher
> layer depends on all packages that are in lower layers.
> Layer 0: BasePkg.
> Layer 1: UefiPkg.
> Layer 2: PiPeiPkg 
> Layer 3: PiDxePkg
> Layer 4: PiMmPkg
> Layer 5: MdeModulePkg (TianoPkg?)
> 
> ####All other modules are put to small packages under Device/ or Feature/.
> 
> ============== Benefit of this proposal =================
> 
> This helps to reduce the size of each package, especially the very BIG
> MdeModulePkg which contains almost all edk2 modules (except
> CPU, network, etc). So platform can use git sparse checkout feature
> to only clone the needed code still in package granularity.
> This also helps to separate the code maintenance to more expert
> developers. MdeModulePkg is just too huge to be maintained by 2 or 3
> developers.

from a first / quick skim, it sounds OK to me; however, I'd like to
explicitly defer on this to the other stewards & stakeholders. I
remember that Leif had ideas about reorganizing the tree.

(Also, I vaguely feel that the movement/renaming of some macros /
definitions that Andrew and Mike have been discussing in thread

  [edk2] History question about Base.h and its alternate parallel name
         space.... Should we change it?

might overlap with this reorg.)

Regarding the benefits, I agree that we need clearer / finer grained
assignments between modules / packages and maintainers. I'm unsure if
that really requires reorganizing the tree (we could just reorganize
Maintainers.txt instead -- add some pathname patterns), but I agree that
reorganizing the tree is one method that could work.

Regarding sparse git checkout -- I'm probably missing details of this
git feature itself (regardless of subject project), but I'm generally
indifferent / unexcited about this. On my disk, a clean QEMU tree is
twice as large as edk2, and a clean Linux tree is more than thrice as
large. Also, it's been years since I decided it was impossible for me to
work without a good SSD (i.e. that traditional spindle disks would be
way too slow.) So, if the reorg helps some developers with handling the
tree locally, I don't mind, but personally I don't consider the reorg a
benefit for that.

Again, I'd like to leave the specifics to Leif, Mike, and others. I hope
that's acceptable.

Thanks
Laszlo

Re: [RFC] Proposal to split Pkgs

[Leif Lindholm]

Hi Laszlo,

Managed to spot this before I sent my own reply :)

On Tue, Jan 29, 2019 at 12:11:51PM +0100, Laszlo Ersek wrote:
> Hi Ray,
> 
> On 01/29/19 06:59, Ni, Ray wrote:
> > Hello,
> > I'd like to propose to split today's BIG packages in following ways:
> > 
> > ==============Overview =================
> > 
> > 1. Separate Industry standard definitions from UEFI and PI interfaces.
> > 2. Separate UEFI and PI interfaces from implementations.
> >     a. Separate UEFI and PI interfaces to different packages
> >     b. Separate PI PEI, DXE and MM phase interfaces to different packages
> > 3. Separate different features into feature packages.
> >     Feature interface may be in the feature package to provide minimal
> >     common interface packages.
> > 
> > The POC code is in https://github.com/jyao1/edk2/tree/ReOrg.
> > It basically split the EDKII common code to three directories:
> > Core/, Device/, and Feature/.
> > The code is in very early POC phase and only code in Core/ directory
> > can pass the build.
> > I would like to gather feedbacks through this RFC to see whether
> > this splitting direction makes sense.
> > Is there any other better ways of splitting?
> > Or perhaps do not split in such a small granularity?
> > Or perhaps Mike's work to move lib-c content to edk2-libc repo,
> > to move real platform code to edk2-platform repo is enough for
> > now?
> > 
> > ==============More explanations =================
> > 
> > ####There are 9 packages inside Core/ directory:
> > 1. BasePkg
> > Contains industry standard definitions (exclude UEFI and PI) and base
> > libraries that non-UEFI and non-PI development can depend on.
> > UEFI or PI development can also depend on this package.
> > 2. UefiPkg
> > Contains UEFI interfaces and libraries that UEFI driver-model driver
> > development can depend on.
> > 3. PiPeiPkg
> > Contains PI interfaces and libraries for PEI phase that PEI module
> > development can depend on.
> > 4. PiDxePkg
> > Contains PI interfaces and libraries for DXE phase that DXE module
> > development can depend on.
> > 5. PiMmPkg
> > Contains PI interfaces and libraries for MM phase that MM/SMM
> > module development can depend on.
> > 6. MdeModulePkg (TianoPkg? Name is still TBD)
> > Contains Tiano implementation specific interfaces and libraries.
> > Developing modules for pure UEFI or PI should not depend on this package.
> > 7. PeiFoundationPkg
> > Contains the PEI foundation modules (PeiCore and DxeIpl) and supported
> > libraries.
> > 8. DxeFoundationPkg
> > Contains the DXE foundation modules (DxeCore and RuntimeDxe) and
> > supported libraries.
> > 9. SmmFoundationPkg
> > Contains the SMM foundation modules (SmmCore, SmmIpl and
> > SmmCommunicationBuffer) and supported libraries.
> > 
> > These packages are positioned in different layers. The package in higher
> > layer depends on all packages that are in lower layers.
> > Layer 0: BasePkg.
> > Layer 1: UefiPkg.
> > Layer 2: PiPeiPkg 
> > Layer 3: PiDxePkg
> > Layer 4: PiMmPkg
> > Layer 5: MdeModulePkg (TianoPkg?)
> > 
> > ####All other modules are put to small packages under Device/ or Feature/.
> > 
> > ============== Benefit of this proposal =================
> > 
> > This helps to reduce the size of each package, especially the very BIG
> > MdeModulePkg which contains almost all edk2 modules (except
> > CPU, network, etc). So platform can use git sparse checkout feature
> > to only clone the needed code still in package granularity.
> > This also helps to separate the code maintenance to more expert
> > developers. MdeModulePkg is just too huge to be maintained by 2 or 3
> > developers.
> 
> from a first / quick skim, it sounds OK to me; however, I'd like to
> explicitly defer on this to the other stewards & stakeholders. I
> remember that Leif had ideas about reorganizing the tree.

Heh. Mainly that it needs reorganising :)
Although I may have brought up Mike's long-term plans to reorganise on
several occasions.

> (Also, I vaguely feel that the movement/renaming of some macros /
> definitions that Andrew and Mike have been discussing in thread
> 
>   [edk2] History question about Base.h and its alternate parallel name
>          space.... Should we change it?

Oh, I've been lax at following the list since I came back off
holiday. This looks like a good thread to get stuck into.

> might overlap with this reorg.)
> 
> Regarding the benefits, I agree that we need clearer / finer grained
> assignments between modules / packages and maintainers. I'm unsure if
> that really requires reorganizing the tree (we could just reorganize
> Maintainers.txt instead -- add some pathname patterns), but I agree that
> reorganizing the tree is one method that could work.

I would like both.
This split won't resolve the issue of defining separate maintainers
for */ARM* and */AARCH64* (and RISCV, ...).

But a proper tree rework would help newcomers. Figuring out what goes
where is non-trivial effort, and the archaic names don't help with this.

> Regarding sparse git checkout -- I'm probably missing details of this
> git feature itself (regardless of subject project), but I'm generally
> indifferent / unexcited about this. On my disk, a clean QEMU tree is
> twice as large as edk2, and a clean Linux tree is more than thrice as
> large. Also, it's been years since I decided it was impossible for me to
> work without a good SSD (i.e. that traditional spindle disks would be
> way too slow.) So, if the reorg helps some developers with handling the
> tree locally, I don't mind, but personally I don't consider the reorg a
> benefit for that.

Yeah, I'm with you.
Although I guess the issue is more to do with poor network connections
than slow checkouts. (Because yes, edk2 is tiny :)  

> Again, I'd like to leave the specifics to Leif, Mike, and others. I hope
> that's acceptable.

No problem.

/
    Leif

Re: [RFC] Proposal to split Pkgs

[Ni, Ray]

> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Laszlo
> Ersek
> Sent: Tuesday, January 29, 2019 7:12 PM
> To: Ni, Ray <ray.ni@intel.com>; edk2-devel@lists.01.org
> Cc: Zimmer, Vincent <vincent.zimmer@intel.com>; Wolman, Ayellet
> <ayellet.wolman@intel.com>; Cetola, Stephano <stephano.cetola@intel.com>;
> Kinney, Michael D <michael.d.kinney@intel.com>
> Subject: Re: [edk2] [RFC] Proposal to split Pkgs
> 
> Hi Ray,
> 
> On 01/29/19 06:59, Ni, Ray wrote:
> > Hello,
> > I'd like to propose to split today's BIG packages in following ways:
> >
> > ==============Overview =================
> >
> > 1. Separate Industry standard definitions from UEFI and PI interfaces.
> > 2. Separate UEFI and PI interfaces from implementations.
> >     a. Separate UEFI and PI interfaces to different packages
> >     b. Separate PI PEI, DXE and MM phase interfaces to different
> > packages 3. Separate different features into feature packages.
> >     Feature interface may be in the feature package to provide minimal
> >     common interface packages.
> >
> > The POC code is in https://github.com/jyao1/edk2/tree/ReOrg.
> > It basically split the EDKII common code to three directories:
> > Core/, Device/, and Feature/.
> > The code is in very early POC phase and only code in Core/ directory
> > can pass the build.
> > I would like to gather feedbacks through this RFC to see whether this
> > splitting direction makes sense.
> > Is there any other better ways of splitting?
> > Or perhaps do not split in such a small granularity?
> > Or perhaps Mike's work to move lib-c content to edk2-libc repo, to
> > move real platform code to edk2-platform repo is enough for now?
> >
> > ==============More explanations =================
> >
> > ####There are 9 packages inside Core/ directory:
> > 1. BasePkg
> > Contains industry standard definitions (exclude UEFI and PI) and base
> > libraries that non-UEFI and non-PI development can depend on.
> > UEFI or PI development can also depend on this package.
> > 2. UefiPkg
> > Contains UEFI interfaces and libraries that UEFI driver-model driver
> > development can depend on.
> > 3. PiPeiPkg
> > Contains PI interfaces and libraries for PEI phase that PEI module
> > development can depend on.
> > 4. PiDxePkg
> > Contains PI interfaces and libraries for DXE phase that DXE module
> > development can depend on.
> > 5. PiMmPkg
> > Contains PI interfaces and libraries for MM phase that MM/SMM module
> > development can depend on.
> > 6. MdeModulePkg (TianoPkg? Name is still TBD) Contains Tiano
> > implementation specific interfaces and libraries.
> > Developing modules for pure UEFI or PI should not depend on this package.
> > 7. PeiFoundationPkg
> > Contains the PEI foundation modules (PeiCore and DxeIpl) and supported
> > libraries.
> > 8. DxeFoundationPkg
> > Contains the DXE foundation modules (DxeCore and RuntimeDxe) and
> > supported libraries.
> > 9. SmmFoundationPkg
> > Contains the SMM foundation modules (SmmCore, SmmIpl and
> > SmmCommunicationBuffer) and supported libraries.
> >
> > These packages are positioned in different layers. The package in
> > higher layer depends on all packages that are in lower layers.
> > Layer 0: BasePkg.
> > Layer 1: UefiPkg.
> > Layer 2: PiPeiPkg
> > Layer 3: PiDxePkg
> > Layer 4: PiMmPkg
> > Layer 5: MdeModulePkg (TianoPkg?)
> >
> > ####All other modules are put to small packages under Device/ or Feature/.
> >
> > ============== Benefit of this proposal =================
> >
> > This helps to reduce the size of each package, especially the very BIG
> > MdeModulePkg which contains almost all edk2 modules (except CPU,
> > network, etc). So platform can use git sparse checkout feature to only
> > clone the needed code still in package granularity.
> > This also helps to separate the code maintenance to more expert
> > developers. MdeModulePkg is just too huge to be maintained by 2 or 3
> > developers.
> 
> from a first / quick skim, it sounds OK to me; however, I'd like to explicitly defer
> on this to the other stewards & stakeholders. I remember that Leif had ideas
> about reorganizing the tree.

That's great to have feedbacks in the early phase.

> 
> (Also, I vaguely feel that the movement/renaming of some macros / definitions
> that Andrew and Mike have been discussing in thread
> 
>   [edk2] History question about Base.h and its alternate parallel name
>          space.... Should we change it?
> 
> might overlap with this reorg.)
> 
> Regarding the benefits, I agree that we need clearer / finer grained assignments
> between modules / packages and maintainers. I'm unsure if that really requires
> reorganizing the tree (we could just reorganize Maintainers.txt instead -- add
> some pathname patterns), but I agree that reorganizing the tree is one method
> that could work.
I agree adding more information in Maintainers.txt works as well.
Thanks for providing the "pathname pattern" idea.
I may do that in parallel to find more maintainers for MdeModulePkg.
Do you agree to have more than 2 maintainers for MdeModulePkg?

> 
> Regarding sparse git checkout -- I'm probably missing details of this git feature
> itself (regardless of subject project), but I'm generally indifferent / unexcited
> about this. On my disk, a clean QEMU tree is twice as large as edk2, and a clean
> Linux tree is more than thrice as large. Also, it's been years since I decided it was
> impossible for me to work without a good SSD (i.e. that traditional spindle disks
> would be way too slow.) So, if the reorg helps some developers with handling
> the tree locally, I don't mind, but personally I don't consider the reorg a benefit
> for that.

Thanks for providing your thoughts about the code size. And the code size data of
QEMU/Linux is helpful.

> 
> Again, I'd like to leave the specifics to Leif, Mike, and others. I hope that's
> acceptable.

Thank you very much for your quick response.

> 
> Thanks
> Laszlo
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: [RFC] Proposal to split Pkgs

[Laszlo Ersek]

On 01/29/19 14:13, Ni, Ray wrote:
> 
> 
>> -----Original Message-----
>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Laszlo
>> Ersek
>> Sent: Tuesday, January 29, 2019 7:12 PM
>> To: Ni, Ray <ray.ni@intel.com>; edk2-devel@lists.01.org
>> Cc: Zimmer, Vincent <vincent.zimmer@intel.com>; Wolman, Ayellet
>> <ayellet.wolman@intel.com>; Cetola, Stephano <stephano.cetola@intel.com>;
>> Kinney, Michael D <michael.d.kinney@intel.com>
>> Subject: Re: [edk2] [RFC] Proposal to split Pkgs

>> Regarding the benefits, I agree that we need clearer / finer grained assignments
>> between modules / packages and maintainers. I'm unsure if that really requires
>> reorganizing the tree (we could just reorganize Maintainers.txt instead -- add
>> some pathname patterns), but I agree that reorganizing the tree is one method
>> that could work.

> I agree adding more information in Maintainers.txt works as well.
> Thanks for providing the "pathname pattern" idea.
> I may do that in parallel to find more maintainers for MdeModulePkg.
> Do you agree to have more than 2 maintainers for MdeModulePkg?

I certainly do; I think we should assign maintainers under MdeModulePkg
on a module (library instance, driver) basis.

Thanks,
Laszlo

Re: [RFC] Proposal to split Pkgs

[Leif Lindholm]

Hi Ray,

First of all, thank you for restarting this discussion, and putting
the effort in for a POC.

On Tue, Jan 29, 2019 at 05:59:35AM +0000, Ni, Ray wrote:
> Hello,
> I'd like to propose to split today's BIG packages in following ways:
> 
> ==============Overview =================
> 
> 1. Separate Industry standard definitions from UEFI and PI interfaces.
> 2. Separate UEFI and PI interfaces from implementations.
>     a. Separate UEFI and PI interfaces to different packages
>     b. Separate PI PEI, DXE and MM phase interfaces to different packages
> 3. Separate different features into feature packages.
>     Feature interface may be in the feature package to provide minimal
>     common interface packages.
> 
> The POC code is in https://github.com/jyao1/edk2/tree/ReOrg.
> It basically split the EDKII common code to three directories:
> Core/, Device/, and Feature/.

I'm not sure I like the 'Device' name, but that's not that important
at this stage - I like that this is split up the way it is.

I also think a lot of things currently under Feture/misc could be
broken out into additional subdirectories under Feature/.

> The code is in very early POC phase and only code in Core/ directory
> can pass the build.
> I would like to gather feedbacks through this RFC to see whether
> this splitting direction makes sense.
> Is there any other better ways of splitting?
> Or perhaps do not split in such a small granularity?

I think this would be my only negative(ish) feedback on the POC.

While it has the benefit of modules requiring to specify more
precisely which functionality they are pulling in...
...it means that they now need to list *everything*.

And with the mechanism by which I have seen Intel engineers make use
of PACKAGES_PATH in the past*, we end up with a bazillion packages
that need to be added individually to this variable for each build.

* (arguably how it was designed to be used - just not how I am
  interested in using it)

It also means that we need *tons* of dummy .dsc files in order to run
through simple build tests.

My preference would be to push the packages back up to the top-level.
The split still makes sense, and allocating maintainers to sub-parts
can happen without a full-out redesign of the Maintainers.txt format.
(Which I will try to resurrect with a proposal over the next few weeks
anyway.)

> Or perhaps Mike's work to move lib-c content to edk2-libc repo,
> to move real platform code to edk2-platform repo is enough for
> now?

I think we need this further restructuring, and renaming. Not just for
splitting up maintainership duties (which as I mentioned in email to
Laszlo, we will still need to do with this change). But to make the
codebase more approachable.

> ==============More explanations =================
> 
> ####There are 9 packages inside Core/ directory:
> 1. BasePkg
> Contains industry standard definitions (exclude UEFI and PI) and base
> libraries that non-UEFI and non-PI development can depend on.
> UEFI or PI development can also depend on this package.
> 2. UefiPkg
> Contains UEFI interfaces and libraries that UEFI driver-model driver
> development can depend on.
> 3. PiPeiPkg
> Contains PI interfaces and libraries for PEI phase that PEI module
> development can depend on.
> 4. PiDxePkg
> Contains PI interfaces and libraries for DXE phase that DXE module
> development can depend on.

Really happy with the above split.

> 5. PiMmPkg
> Contains PI interfaces and libraries for MM phase that MM/SMM
> module development can depend on.

How would/should this work relative to StandaloneMmPkg?

> 6. MdeModulePkg (TianoPkg? Name is still TBD)

Yes, this needs to be renamed. I don't object to the Tiano naming.

> Contains Tiano implementation specific interfaces and libraries.
> Developing modules for pure UEFI or PI should not depend on this package.
> 7. PeiFoundationPkg
> Contains the PEI foundation modules (PeiCore and DxeIpl) and supported
> libraries.
> 8. DxeFoundationPkg
> Contains the DXE foundation modules (DxeCore and RuntimeDxe) and
> supported libraries.
> 9. SmmFoundationPkg
> Contains the SMM foundation modules (SmmCore, SmmIpl and
> SmmCommunicationBuffer) and supported libraries.

So as a feedback on the whole, this would somewhat help with
addressing the confusion caused by PI defining interfaces named EFI_*
(which we really ought to address on the spec level at some point).

> These packages are positioned in different layers. The package in higher
> layer depends on all packages that are in lower layers.
> Layer 0: BasePkg.
> Layer 1: UefiPkg.

This looks a bit weird though - surely PI components should not have
dependencies on UEFI components? I presume this is what Laszlo
referred to with regards to Base.h and the thread from earlier this
month.

> Layer 2: PiPeiPkg 
> Layer 3: PiDxePkg
> Layer 4: PiMmPkg
> Layer 5: MdeModulePkg (TianoPkg?)
> 
> ####All other modules are put to small packages under Device/ or Feature/.
> 
> ============== Benefit of this proposal =================
> 
> This helps to reduce the size of each package, especially the very BIG
> MdeModulePkg which contains almost all edk2 modules (except
> CPU, network, etc). So platform can use git sparse checkout feature
> to only clone the needed code still in package granularity.
> This also helps to separate the code maintenance to more expert
> developers. MdeModulePkg is just too huge to be maintained by 2 or 3
> developers.

So, I technically don't care about any of the above, but I still
support this proposal (given some additional tweaks). To me, this is
all about reflecing boundaries between specification and
implementation, as well as boundaries between specfications.

Best Regards,

Leif

Re: [RFC] Proposal to split Pkgs

[Ni, Ray]

On 1/29/2019 10:21 PM, Leif Lindholm wrote:
> Hi Ray,
> 
> First of all, thank you for restarting this discussion, and putting
> the effort in for a POC.
> 
> On Tue, Jan 29, 2019 at 05:59:35AM +0000, Ni, Ray wrote:
>> Hello,
>> I'd like to propose to split today's BIG packages in following ways:
>>
>> ==============Overview =================
>>
>> 1. Separate Industry standard definitions from UEFI and PI interfaces.
>> 2. Separate UEFI and PI interfaces from implementations.
>>      a. Separate UEFI and PI interfaces to different packages
>>      b. Separate PI PEI, DXE and MM phase interfaces to different packages
>> 3. Separate different features into feature packages.
>>      Feature interface may be in the feature package to provide minimal
>>      common interface packages.
>>
>> The POC code is in https://github.com/jyao1/edk2/tree/ReOrg.
>> It basically split the EDKII common code to three directories:
>> Core/, Device/, and Feature/.
> 
> I'm not sure I like the 'Device' name, but that's not that important
> at this stage - I like that this is split up the way it is.
> 
> I also think a lot of things currently under Feture/misc could be
> broken out into additional subdirectories under Feature/.
> 
>> The code is in very early POC phase and only code in Core/ directory
>> can pass the build.
>> I would like to gather feedbacks through this RFC to see whether
>> this splitting direction makes sense.
>> Is there any other better ways of splitting?
>> Or perhaps do not split in such a small granularity?
> 
> I think this would be my only negative(ish) feedback on the POC.
> 
> While it has the benefit of modules requiring to specify more
> precisely which functionality they are pulling in...
> ...it means that they now need to list *everything*.
> 
> And with the mechanism by which I have seen Intel engineers make use
> of PACKAGES_PATH in the past*, we end up with a bazillion packages
> that need to be added individually to this variable for each build.

No. "PACKAGES_PATH" only contains paths pointing to the *parent* 
directory of *Pkg. In this case, it equals to:
/work/edk2/Core:/work/edk2/Device:/work/edk2/Feature/security:/work/edk2/Feature/network:/work/edk2/Feature/misc


> 
> * (arguably how it was designed to be used - just not how I am
>    interested in using it)
> 
> It also means that we need *tons* of dummy .dsc files in order to run
> through simple build tests.

Yes each small package needs its own DSC and DEC.
DSC to list all components.
DEC to list all interfaces.

OK maybe the "need" in your wording means the build test script file 
needs to reference all DSC files.
That maybe can be solved by an advanced script to call "build" for each 
DSC files inside a directory that matches a specific pattern. So the 
build test script to build PiPeiPkg/PiDxePkg/PiMmPkg will be like:
   build-any-dsc /work/edk2/Core/ --pattern="*Pi*"

> 
> My preference would be to push the packages back up to the top-level.
> The split still makes sense, and allocating maintainers to sub-parts
> can happen without a full-out redesign of the Maintainers.txt format.
> (Which I will try to resurrect with a proposal over the next few weeks
> anyway.)

Combining all packages inside the Device and Feature directory to still
one single big package but with another directory layout so that only 
single DSC/DEC is needed?

> 
>> Or perhaps Mike's work to move lib-c content to edk2-libc repo,
>> to move real platform code to edk2-platform repo is enough for
>> now?
> 
> I think we need this further restructuring, and renaming. Not just for
> splitting up maintainership duties (which as I mentioned in email to
> Laszlo, we will still need to do with this change). But to make the
> codebase more approachable.
> 

"more approachable" means more friendly to new comers? Just try to 
understand everyone's real concern to today's codebase.

>> ==============More explanations =================
>>
>> ####There are 9 packages inside Core/ directory:
>> 1. BasePkg
>> Contains industry standard definitions (exclude UEFI and PI) and base
>> libraries that non-UEFI and non-PI development can depend on.
>> UEFI or PI development can also depend on this package.
>> 2. UefiPkg
>> Contains UEFI interfaces and libraries that UEFI driver-model driver
>> development can depend on.
>> 3. PiPeiPkg
>> Contains PI interfaces and libraries for PEI phase that PEI module
>> development can depend on.
>> 4. PiDxePkg
>> Contains PI interfaces and libraries for DXE phase that DXE module
>> development can depend on.
> 
> Really happy with the above split.
> 
>> 5. PiMmPkg
>> Contains PI interfaces and libraries for MM phase that MM/SMM
>> module development can depend on.
> 
> How would/should this work relative to StandaloneMmPkg?
The "Mm" is to reflect to the same word in PI spec.
But I didn't check whether "StandaloneMmPkg" can work with the this.

> 
>> 6. MdeModulePkg (TianoPkg? Name is still TBD)
> 
> Yes, this needs to be renamed. I don't object to the Tiano naming.
> 

Thanks.

>> Contains Tiano implementation specific interfaces and libraries.
>> Developing modules for pure UEFI or PI should not depend on this package.
>> 7. PeiFoundationPkg
>> Contains the PEI foundation modules (PeiCore and DxeIpl) and supported
>> libraries.
>> 8. DxeFoundationPkg
>> Contains the DXE foundation modules (DxeCore and RuntimeDxe) and
>> supported libraries.
>> 9. SmmFoundationPkg
>> Contains the SMM foundation modules (SmmCore, SmmIpl and
>> SmmCommunicationBuffer) and supported libraries.
> 
> So as a feedback on the whole, this would somewhat help with
> addressing the confusion caused by PI defining interfaces named EFI_*
> (which we really ought to address on the spec level at some point).
> 
>> These packages are positioned in different layers. The package in higher
>> layer depends on all packages that are in lower layers.
>> Layer 0: BasePkg.
>> Layer 1: UefiPkg.
> 
> This looks a bit weird though - surely PI components should not have
> dependencies on UEFI components? I presume this is what Laszlo
> referred to with regards to Base.h and the thread from earlier this
> month.

I will provide more data on how the dependency is there.
For now, I know that "EFI_GUID" is defined in UefiPkg. But "EFI_GUID" is 
needed everywhere.

> 
>> Layer 2: PiPeiPkg
>> Layer 3: PiDxePkg
>> Layer 4: PiMmPkg
>> Layer 5: MdeModulePkg (TianoPkg?)
>>
>> ####All other modules are put to small packages under Device/ or Feature/.
>>
>> ============== Benefit of this proposal =================
>>
>> This helps to reduce the size of each package, especially the very BIG
>> MdeModulePkg which contains almost all edk2 modules (except
>> CPU, network, etc). So platform can use git sparse checkout feature
>> to only clone the needed code still in package granularity.
>> This also helps to separate the code maintenance to more expert
>> developers. MdeModulePkg is just too huge to be maintained by 2 or 3
>> developers.
> 
> So, I technically don't care about any of the above, but I still
> support this proposal (given some additional tweaks). To me, this is
> all about reflecting boundaries between specification and
> implementation, as well as boundaries between specifications.

Good to know what you care.

> 
> Best Regards,
> 
> Leif
> 


-- 
Thanks,
Ray

Re: [RFC] Proposal to split Pkgs

[Ard Biesheuvel]

On Thu, 31 Jan 2019 at 08:09, Ni, Ray <ray.ni@intel.com> wrote:
>
> On 1/29/2019 10:21 PM, Leif Lindholm wrote:
> > Hi Ray,
> >
> > First of all, thank you for restarting this discussion, and putting
> > the effort in for a POC.
> >
> > On Tue, Jan 29, 2019 at 05:59:35AM +0000, Ni, Ray wrote:
> >> Hello,
> >> I'd like to propose to split today's BIG packages in following ways:
> >>
> >> ==============Overview =================
> >>
> >> 1. Separate Industry standard definitions from UEFI and PI interfaces.
> >> 2. Separate UEFI and PI interfaces from implementations.
> >>      a. Separate UEFI and PI interfaces to different packages
> >>      b. Separate PI PEI, DXE and MM phase interfaces to different packages
> >> 3. Separate different features into feature packages.
> >>      Feature interface may be in the feature package to provide minimal
> >>      common interface packages.
> >>
> >> The POC code is in https://github.com/jyao1/edk2/tree/ReOrg.
> >> It basically split the EDKII common code to three directories:
> >> Core/, Device/, and Feature/.
> >
> > I'm not sure I like the 'Device' name, but that's not that important
> > at this stage - I like that this is split up the way it is.
> >
> > I also think a lot of things currently under Feture/misc could be
> > broken out into additional subdirectories under Feature/.
> >
> >> The code is in very early POC phase and only code in Core/ directory
> >> can pass the build.
> >> I would like to gather feedbacks through this RFC to see whether
> >> this splitting direction makes sense.
> >> Is there any other better ways of splitting?
> >> Or perhaps do not split in such a small granularity?
> >
> > I think this would be my only negative(ish) feedback on the POC.
> >
> > While it has the benefit of modules requiring to specify more
> > precisely which functionality they are pulling in...
> > ...it means that they now need to list *everything*.
> >
> > And with the mechanism by which I have seen Intel engineers make use
> > of PACKAGES_PATH in the past*, we end up with a bazillion packages
> > that need to be added individually to this variable for each build.
>
> No. "PACKAGES_PATH" only contains paths pointing to the *parent*
> directory of *Pkg. In this case, it equals to:
> /work/edk2/Core:/work/edk2/Device:/work/edk2/Feature/security:/work/edk2/Feature/network:/work/edk2/Feature/misc
>

Could we *please* not do this? PACKAGES_PATH is very useful for
combining packages from different repositories, but that does not mean
we should split up the core repository in this way.

What we will end up with is platforms that use

  Core/Package/Foo.inf

and platforms that use

  Package/Foo.inf

which creates an aliasing mess where the same file is reachable in
different ways. Also, it encourages the behavior where you need to
fetch version X of repo A and version Y of repo B, and the whole point
of open sourcing is to have all code co-exist in the same source tree.





>
> >
> > * (arguably how it was designed to be used - just not how I am
> >    interested in using it)
> >
> > It also means that we need *tons* of dummy .dsc files in order to run
> > through simple build tests.
>
> Yes each small package needs its own DSC and DEC.
> DSC to list all components.
> DEC to list all interfaces.
>
> OK maybe the "need" in your wording means the build test script file
> needs to reference all DSC files.
> That maybe can be solved by an advanced script to call "build" for each
> DSC files inside a directory that matches a specific pattern. So the
> build test script to build PiPeiPkg/PiDxePkg/PiMmPkg will be like:
>    build-any-dsc /work/edk2/Core/ --pattern="*Pi*"
>
> >
> > My preference would be to push the packages back up to the top-level.
> > The split still makes sense, and allocating maintainers to sub-parts
> > can happen without a full-out redesign of the Maintainers.txt format.
> > (Which I will try to resurrect with a proposal over the next few weeks
> > anyway.)
>
> Combining all packages inside the Device and Feature directory to still
> one single big package but with another directory layout so that only
> single DSC/DEC is needed?
>
> >
> >> Or perhaps Mike's work to move lib-c content to edk2-libc repo,
> >> to move real platform code to edk2-platform repo is enough for
> >> now?
> >
> > I think we need this further restructuring, and renaming. Not just for
> > splitting up maintainership duties (which as I mentioned in email to
> > Laszlo, we will still need to do with this change). But to make the
> > codebase more approachable.
> >
>
> "more approachable" means more friendly to new comers? Just try to
> understand everyone's real concern to today's codebase.
>
> >> ==============More explanations =================
> >>
> >> ####There are 9 packages inside Core/ directory:
> >> 1. BasePkg
> >> Contains industry standard definitions (exclude UEFI and PI) and base
> >> libraries that non-UEFI and non-PI development can depend on.
> >> UEFI or PI development can also depend on this package.
> >> 2. UefiPkg
> >> Contains UEFI interfaces and libraries that UEFI driver-model driver
> >> development can depend on.
> >> 3. PiPeiPkg
> >> Contains PI interfaces and libraries for PEI phase that PEI module
> >> development can depend on.
> >> 4. PiDxePkg
> >> Contains PI interfaces and libraries for DXE phase that DXE module
> >> development can depend on.
> >
> > Really happy with the above split.
> >
> >> 5. PiMmPkg
> >> Contains PI interfaces and libraries for MM phase that MM/SMM
> >> module development can depend on.
> >
> > How would/should this work relative to StandaloneMmPkg?
> The "Mm" is to reflect to the same word in PI spec.
> But I didn't check whether "StandaloneMmPkg" can work with the this.
>
> >
> >> 6. MdeModulePkg (TianoPkg? Name is still TBD)
> >
> > Yes, this needs to be renamed. I don't object to the Tiano naming.
> >
>
> Thanks.
>
> >> Contains Tiano implementation specific interfaces and libraries.
> >> Developing modules for pure UEFI or PI should not depend on this package.
> >> 7. PeiFoundationPkg
> >> Contains the PEI foundation modules (PeiCore and DxeIpl) and supported
> >> libraries.
> >> 8. DxeFoundationPkg
> >> Contains the DXE foundation modules (DxeCore and RuntimeDxe) and
> >> supported libraries.
> >> 9. SmmFoundationPkg
> >> Contains the SMM foundation modules (SmmCore, SmmIpl and
> >> SmmCommunicationBuffer) and supported libraries.
> >
> > So as a feedback on the whole, this would somewhat help with
> > addressing the confusion caused by PI defining interfaces named EFI_*
> > (which we really ought to address on the spec level at some point).
> >
> >> These packages are positioned in different layers. The package in higher
> >> layer depends on all packages that are in lower layers.
> >> Layer 0: BasePkg.
> >> Layer 1: UefiPkg.
> >
> > This looks a bit weird though - surely PI components should not have
> > dependencies on UEFI components? I presume this is what Laszlo
> > referred to with regards to Base.h and the thread from earlier this
> > month.
>
> I will provide more data on how the dependency is there.
> For now, I know that "EFI_GUID" is defined in UefiPkg. But "EFI_GUID" is
> needed everywhere.
>
> >
> >> Layer 2: PiPeiPkg
> >> Layer 3: PiDxePkg
> >> Layer 4: PiMmPkg
> >> Layer 5: MdeModulePkg (TianoPkg?)
> >>
> >> ####All other modules are put to small packages under Device/ or Feature/.
> >>
> >> ============== Benefit of this proposal =================
> >>
> >> This helps to reduce the size of each package, especially the very BIG
> >> MdeModulePkg which contains almost all edk2 modules (except
> >> CPU, network, etc). So platform can use git sparse checkout feature
> >> to only clone the needed code still in package granularity.
> >> This also helps to separate the code maintenance to more expert
> >> developers. MdeModulePkg is just too huge to be maintained by 2 or 3
> >> developers.
> >
> > So, I technically don't care about any of the above, but I still
> > support this proposal (given some additional tweaks). To me, this is
> > all about reflecting boundaries between specification and
> > implementation, as well as boundaries between specifications.
>
> Good to know what you care.
>
> >
> > Best Regards,
> >
> > Leif
> >
>
>
> --
> Thanks,
> Ray
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: [RFC] Proposal to split Pkgs

[Ni, Ray]

On 1/31/2019 3:51 PM, Ard Biesheuvel wrote:
>>>> The POC code is inhttps://github.com/jyao1/edk2/tree/ReOrg.
>>>> It basically split the EDKII common code to three directories:
>>>> Core/, Device/, and Feature/.
>>> I'm not sure I like the 'Device' name, but that's not that important
>>> at this stage - I like that this is split up the way it is.
>>>
>>> I also think a lot of things currently under Feture/misc could be
>>> broken out into additional subdirectories under Feature/.
>>>
>>>> The code is in very early POC phase and only code in Core/ directory
>>>> can pass the build.
>>>> I would like to gather feedbacks through this RFC to see whether
>>>> this splitting direction makes sense.
>>>> Is there any other better ways of splitting?
>>>> Or perhaps do not split in such a small granularity?
>>> I think this would be my only negative(ish) feedback on the POC.
>>>
>>> While it has the benefit of modules requiring to specify more
>>> precisely which functionality they are pulling in...
>>> ...it means that they now need to list*everything*.
>>>
>>> And with the mechanism by which I have seen Intel engineers make use
>>> of PACKAGES_PATH in the past*, we end up with a bazillion packages
>>> that need to be added individually to this variable for each build.
>> No. "PACKAGES_PATH" only contains paths pointing to the*parent*
>> directory of *Pkg. In this case, it equals to:
>> /work/edk2/Core:/work/edk2/Device:/work/edk2/Feature/security:/work/edk2/Feature/network:/work/edk2/Feature/misc
>>
> Could we*please*  not do this? PACKAGES_PATH is very useful for
> combining packages from different repositories, but that does not mean
> we should split up the core repository in this way.
> 
> What we will end up with is platforms that use
> 
>    Core/Package/Foo.inf
> 
> and platforms that use
> 
>    Package/Foo.inf
> 
> which creates an aliasing mess where the same file is reachable in
> different ways. Also, it encourages the behavior where you need to
> fetch version X of repo A and version Y of repo B, and the whole point
> of open sourcing is to have all code co-exist in the same source tree.
> 
> 
Ard,
Some platforms which define PACKAGES_PATH to "/work/edk2" will have 
"Core/Package/Foo.inf" in the DSC/FDF.
Some platforms which define PACKAGES_PATH to "/work/edk2/Core" will have 
"Package/Foo.inf" in the DSC/FDF.
It's bad and I 100% agree!
The basic rule behind that is PACKAGES_PATH should only contains path 
pointing to the root of repo.

Thanks for raising a very good concern on putting packages inside 
sub-directory.


-- 
Thanks,
Ray

Re: [RFC] Proposal to split Pkgs

[Andrew Fish]

> On Jan 28, 2019, at 9:59 PM, Ni, Ray <ray.ni@intel.com> wrote:
> 
> Hello,
> I'd like to propose to split today's BIG packages in following ways:
> 
> ==============Overview =================
> 
> 1. Separate Industry standard definitions from UEFI and PI interfaces.

Ray,

>From a big picture point of view lets talk about customer impact....

Splitting the MdePkg means:
1) Every INF file needs a new  [Packages] layout, so has to change. 
2) Every DSC file needs to update [LibraryClasses] pathing.

Moving the drivers around:
1) Every DSC file has to change to update the path to the driver
2) Every FDF file has to change to update the path to the driver. 

I'd point out forcing every 3rd party module (library, driver, or application) to change is a much bigger impact that forcing a change to a projects target platform (DSC, FDF). 

I'm trying to figure out how to make a common code base that spans several generations of vendor code. So obviously today every module depends of MdePkg, and each new product is going to require new drivers for the chipset and peripherals. So lets say I'm a 3rd party graphics vendor does this mean I need and edk2 version of an INF and an edk3 version (no more MdePkg) of the INF as my customers are going to move at different rates? On the flip side how do I make a common platform tree if my vendor for work station chipsets lags behind my other chipset vendor in regards to moving to edk3. Does that mean I have to port that vendors code to edk3, which does not sound bad until you realize that every code drop from the vendor I need to remerge my edk3 changes back on top of it. Seems like a lot of work. 

Thus I think we need to ask the question do we need to leave the MdePkg around for compatibility? Do we need an INF syntax that supports edk2 and edk3 (Like the #ifdef we had that supported EDK and edk2 includes back in the day). We could have edk2 and edk3 INF files for very module?  Do we need to build a synthetic MdePkg that gets you the correct packages paths for the new include layout? I guess we could just make MdePkg an alias in the INF file for the 3 (?) new packages? Or do we not change all the include paths?

Seems like we are creating our own Python 2.* vs. 3.* mess to make it easier to maintain the packages? I'm not saying it is wrong, but we should ask the question how is going to impact the edk2 consumers. 

Thanks,

Andrew Fish

> 2. Separate UEFI and PI interfaces from implementations.
>    a. Separate UEFI and PI interfaces to different packages
>    b. Separate PI PEI, DXE and MM phase interfaces to different packages
> 3. Separate different features into feature packages.
>    Feature interface may be in the feature package to provide minimal
>    common interface packages.
> 
> The POC code is in https://github.com/jyao1/edk2/tree/ReOrg.
> It basically split the EDKII common code to three directories:
> Core/, Device/, and Feature/.
> The code is in very early POC phase and only code in Core/ directory
> can pass the build.
> I would like to gather feedbacks through this RFC to see whether
> this splitting direction makes sense.
> Is there any other better ways of splitting?
> Or perhaps do not split in such a small granularity?
> Or perhaps Mike's work to move lib-c content to edk2-libc repo,
> to move real platform code to edk2-platform repo is enough for
> now?
> 
> ==============More explanations =================
> 
> ####There are 9 packages inside Core/ directory:
> 1. BasePkg
> Contains industry standard definitions (exclude UEFI and PI) and base
> libraries that non-UEFI and non-PI development can depend on.
> UEFI or PI development can also depend on this package.
> 2. UefiPkg
> Contains UEFI interfaces and libraries that UEFI driver-model driver
> development can depend on.
> 3. PiPeiPkg
> Contains PI interfaces and libraries for PEI phase that PEI module
> development can depend on.
> 4. PiDxePkg
> Contains PI interfaces and libraries for DXE phase that DXE module
> development can depend on.
> 5. PiMmPkg
> Contains PI interfaces and libraries for MM phase that MM/SMM
> module development can depend on.
> 6. MdeModulePkg (TianoPkg? Name is still TBD)
> Contains Tiano implementation specific interfaces and libraries.
> Developing modules for pure UEFI or PI should not depend on this package.
> 7. PeiFoundationPkg
> Contains the PEI foundation modules (PeiCore and DxeIpl) and supported
> libraries.
> 8. DxeFoundationPkg
> Contains the DXE foundation modules (DxeCore and RuntimeDxe) and
> supported libraries.
> 9. SmmFoundationPkg
> Contains the SMM foundation modules (SmmCore, SmmIpl and
> SmmCommunicationBuffer) and supported libraries.
> 
> These packages are positioned in different layers. The package in higher
> layer depends on all packages that are in lower layers.
> Layer 0: BasePkg.
> Layer 1: UefiPkg.
> Layer 2: PiPeiPkg 
> Layer 3: PiDxePkg
> Layer 4: PiMmPkg
> Layer 5: MdeModulePkg (TianoPkg?)
> 
> ####All other modules are put to small packages under Device/ or Feature/.
> 
> ============== Benefit of this proposal =================
> 
> This helps to reduce the size of each package, especially the very BIG
> MdeModulePkg which contains almost all edk2 modules (except
> CPU, network, etc). So platform can use git sparse checkout feature
> to only clone the needed code still in package granularity.
> This also helps to separate the code maintenance to more expert
> developers. MdeModulePkg is just too huge to be maintained by 2 or 3
> developers.
> 
> Thanks,
> Ray
> <winmail.dat>