GSOC 2021 EXT4 driver Project

GSOC 2021 EXT4 driver Project

[Pedro Falcato]

[-- Attachment #1: Type: text/plain, Size: 1274 bytes --]

Hi everyone,

Me and my project have been selected for GSoC this year, under Michael Kinney and bret. Thank you for the opportunity to collaborate with you and improve Tianocore!
If anyone has any questions, please fire away :)
How do I get started? I'd like to find some easier tasks as to start trying out patch submission and generally programming in a firmware environment.
Also, I've been talking with my mentors and a relevant question to ask the mailing list is: Where should we put the EXT4 driver?
Michael said there are other filesystems in MdeModulePkg, but it might be getting too big and proposed the following options:

1) EXT4 in new package in edk2 repo as a peer to FatPkg.
2) EXT4 in edk2 repo in MdeModulePkg
3) EXT4 in edk2-platforms advanced feature package.
4) EXT4 in edk2 advanced feature package

As someone that's still learning how to navigate the project's tree(s), this is a bit over my head and so I'd like your opinion on the matter.
Also, I would love if someone could point me to some good reading material and/or examples of the package/build system, as I couldn't find documentation on those
and my previous experiment with Tianocore involved looking at FatPkg and mindlessly copying what it was doing.

Thanks,

Pedro

[-- Attachment #2: Type: text/html, Size: 2003 bytes --]

Re: [edk2-devel] GSOC 2021 EXT4 driver Project

[Michael D Kinney]

Hi Pedro,

Here is a link to background materials to get started with EDK II.

    https://github.com/tianocore-training/Tianocore_Training_Contents/wiki

The detailed specifications for the EDK II build systems and coding style are:

    https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Draft-Specification

Best regards,

Mike

===================

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Pedro Falcato
Sent: Monday, May 24, 2021 12:27 PM
To: devel@edk2.groups.io
Subject: [edk2-devel] GSOC 2021 EXT4 driver Project

Hi everyone,

Me and my project have been selected for GSoC this year, under Michael Kinney and bret. Thank you for the opportunity to collaborate with you and improve Tianocore!
If anyone has any questions, please fire away :)
How do I get started? I'd like to find some easier tasks as to start trying out patch submission and generally programming in a firmware environment.
Also, I've been talking with my mentors and a relevant question to ask the mailing list is: Where should we put the EXT4 driver?
Michael said there are other filesystems in MdeModulePkg, but it might be getting too big and proposed the following options:

1) EXT4 in new package in edk2 repo as a peer to FatPkg.
2) EXT4 in edk2 repo in MdeModulePkg
3) EXT4 in edk2-platforms advanced feature package.
4) EXT4 in edk2 advanced feature package

As someone that's still learning how to navigate the project's tree(s), this is a bit over my head and so I'd like your opinion on the matter.
Also, I would love if someone could point me to some good reading material and/or examples of the package/build system, as I couldn't find documentation on those
and my previous experiment with Tianocore involved looking at FatPkg and mindlessly copying what it was doing.

Thanks,

Pedro  

Re: [edk2-devel] GSOC 2021 EXT4 driver Project

[Michael Brown]

On 24/05/2021 20:26, Pedro Falcato wrote:
> Me and my project have been selected for GSoC this year, under Michael 
> Kinney and bret. Thank you for the opportunity to collaborate with you 
> and improve Tianocore!
> If anyone has any questions, please fire away :)
> How do I get started? I'd like to find some easier tasks as to start 
> trying out patch submission and generally programming in a firmware 
> environment.
> Also, I've been talking with my mentors and a relevant question to ask 
> the mailing list is: Where should we put the EXT4 driver?
> Michael said there are other filesystems in MdeModulePkg, but it might 
> be getting too big and proposed the following options:
> 
> 1) EXT4 in new package in edk2 repo as a peer to FatPkg.
> 2) EXT4 in edk2 repo in MdeModulePkg
> 3) EXT4 in edk2-platforms advanced feature package.
> 4) EXT4 in edk2 advanced feature package

FWIW, the most natural positioning seems to me to be as a separate 
Ext4Pkg similar to FatPkg.

For the sake of your own sanity, you probably want to start by 
maintaining Ext4Pkg as a completely standalone git repo that you happen 
to place within your local edk2 checkout directory.  This will let you 
develop in isolation without worrying about where the code will 
eventually live, at least until the code has reached a state of maturity 
that makes it worth considering its eventual home.  Keeping it in a 
separate Ext4Pkg repo will also enforce some level of discipline on your 
coding: you'll naturally avoid being tempted into letting your 
development work leak outside of Ext4Pkg into other packages.

In terms of questions, there are two that immediately spring to mind:

- Are you planning on making this read-only, or read-write?

- If you're planning on making it read-write: what simplifications will 
you be able to make to the write path in the UEFI environment?  (UEFI is 
basically a single-user environment in which all filesystem writes 
should be flushed immediately: you may want to consider a simplified 
write path that never uses journalling, for example).

HTH,

Michael

Re: [edk2-devel] GSOC 2021 EXT4 driver Project

[Pedro Falcato]

[-- Attachment #1: Type: text/plain, Size: 1867 bytes --]

Hi Michael,

Sorry for the delay, I've been quite busy with work these past few days.

Yes, it sounds good, I'll keep it on a separate repo until it's actually more or less mature.

As for your questions,

1) I'm planning to make this read-write, but most importantly, I want it to be solid. If I can't reach write support, it's okay, since it's just a stretch goal.
My idea was to get solid read-only support, as it's naturally the most important part of a firmware filesystem driver (and as far as I know, there are not many
EFI applications/bootloaders that write to the filesystem). If I get the desired solidity and unit tests going, I want to make it write-capable too.

2) Although I'd love to avoid journaling, which is a matter I'm not too familiar with, I'm not entirely sure what simplifications may be done because for one,
what happens if the power cuts during a write? It's unclear to me how a FW filesystem driver might work on a damaged filesystem like that, since it's not at all similar to an OS,
which usually can do some sort of 'fsck' invocation to repair a filesystem before it's mounted. Is the firmware essentially unable to boot to those partitions until
someone gets a recovery drive of some sort that has a 'fsck' on it? I hope the FAT32 code has some answers for me, but I haven't had the time to go look at it that closely just yet.
It might be that the chance of this happening is minimal, but that doesn't sit right with me.

Also, one question: Does firmware code need the usual synchronization primitives (only spinlocks in this case, I would assume) or is it just assumed that it's a single threaded
environment? I know UEFI doesn't have threads but there are places in code that use things like EFI_MP_SERVICES, can the APs never touch certain code (like filesystem code, for example)?

Best Regards,

Pedro

[-- Attachment #2: Type: text/html, Size: 1959 bytes --]

Re: [edk2-devel] GSOC 2021 EXT4 driver Project

[Michael Brown]

On 28/05/2021 18:16, Pedro Falcato wrote:
> 2) Although I'd love to avoid journaling, which is a matter I'm not too 
> familiar with, I'm not entirely sure what simplifications may be done 
> because for one,
> what happens if the power cuts during a write? It's unclear to me how a 
> FW filesystem driver might work on a damaged filesystem like that, since 
> it's not at all similar to an OS,
> which usually can do some sort of 'fsck' invocation to repair a 
> filesystem before it's mounted. Is the firmware essentially unable to 
> boot to those partitions until
> someone gets a recovery drive of some sort that has a 'fsck' on it? I 
> hope the FAT32 code has some answers for me, but I haven't had the time 
> to go look at it that closely just yet.
> It might be that the chance of this happening is minimal, but that 
> doesn't sit right with me.

I don't know the internals of Ext4 journalling well enough to comment in 
detail, but my guess is that you are likely to find that some aspects 
are required for correctness but some aspects are required only for fast 
write performance with multiple concurrent processes (which is 
completely irrelevant for boot firmware).

> Also, one question: Does firmware code need the usual synchronization 
> primitives (only spinlocks in this case, I would assume) or is it just 
> assumed that it's a single threaded
> environment? I know UEFI doesn't have threads but there are places in 
> code that use things like EFI_MP_SERVICES, can the APs never touch 
> certain code (like filesystem code, for example)?

UEFI is fundamentally single-process, single-threaded, and based on 
polling rather than interrupts.  It is _almost_ a clean design in which 
code never needs to worry about locking or other forms of synchronization.

Unfortunately this design is compromised by the existence of UEFI 
timers.  There is no way to hook in a useful hardware interrupt (e.g. 
for a NIC received packet), but there are timer interrupts that will 
fire at unpredictable times and can result in arbitrary callbacks being 
invoked.

This introduces a requirement for some kind of synchronization, which 
UEFI handles via RaiseTPL()/RestoreTPL().  You can use RaiseTPL() to 
effectively disable timer interrupts and thereby guarantee that your 
code will not be reentered.

There is essentially no formal specification of what code should be 
allowed to run at each TPL, so your only viable option is to dig through 
existing EDK2 implementations to infer the de facto requirements.

Michael

Re: [edk2-devel] GSOC 2021 EXT4 driver Project

[Michael D Kinney]

[-- Attachment #1: Type: text/plain, Size: 1958 bytes --]

Hi Pedro,

After thinking about this, I think I would prefer Option (4).

The proposed landing zone would be a new Ext4Pkg in the edk2-platforms repository in the Features directory.

edk2-platforms/Features/Intel/Ext4Pkg

All the features in that directory and not specific to Intel.  There are other email discussions about moving some of that content up a level, so an alternative path would be:

              edk2-platforms/Features/Ext4Pkg

Best regards,

Mike


From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Pedro Falcato
Sent: Monday, May 24, 2021 12:27 PM
To: devel@edk2.groups.io
Subject: [edk2-devel] GSOC 2021 EXT4 driver Project

Hi everyone,

Me and my project have been selected for GSoC this year, under Michael Kinney and bret. Thank you for the opportunity to collaborate with you and improve Tianocore!
If anyone has any questions, please fire away :)
How do I get started? I'd like to find some easier tasks as to start trying out patch submission and generally programming in a firmware environment.
Also, I've been talking with my mentors and a relevant question to ask the mailing list is: Where should we put the EXT4 driver?
Michael said there are other filesystems in MdeModulePkg, but it might be getting too big and proposed the following options:

1) EXT4 in new package in edk2 repo as a peer to FatPkg.
2) EXT4 in edk2 repo in MdeModulePkg
3) EXT4 in edk2-platforms advanced feature package.
4) EXT4 in edk2 advanced feature package

As someone that's still learning how to navigate the project's tree(s), this is a bit over my head and so I'd like your opinion on the matter.
Also, I would love if someone could point me to some good reading material and/or examples of the package/build system, as I couldn't find documentation on those
and my previous experiment with Tianocore involved looking at FatPkg and mindlessly copying what it was doing.

Thanks,

Pedro


[-- Attachment #2: Type: text/html, Size: 43195 bytes --]

Re: [edk2-devel] GSOC 2021 EXT4 driver Project

[Leif Lindholm]

edk2-platforms/Features/Ext4Pkg sounds good to me too.

/
    Leif

On Thu, Jun 10, 2021 at 17:38:17 +0000, Michael D Kinney wrote:
> Hi Pedro,
> 
> After thinking about this, I think I would prefer Option (4).
> 
> The proposed landing zone would be a new Ext4Pkg in the edk2-platforms repository in the Features directory.
> 
> edk2-platforms/Features/Intel/Ext4Pkg
> 
> All the features in that directory and not specific to Intel.  There are other email discussions about moving some of that content up a level, so an alternative path would be:
> 
>               edk2-platforms/Features/Ext4Pkg
> 
> Best regards,
> 
> Mike
> 
> 
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Pedro Falcato
> Sent: Monday, May 24, 2021 12:27 PM
> To: devel@edk2.groups.io
> Subject: [edk2-devel] GSOC 2021 EXT4 driver Project
> 
> Hi everyone,
> 
> Me and my project have been selected for GSoC this year, under Michael Kinney and bret. Thank you for the opportunity to collaborate with you and improve Tianocore!
> If anyone has any questions, please fire away :)
> How do I get started? I'd like to find some easier tasks as to start trying out patch submission and generally programming in a firmware environment.
> Also, I've been talking with my mentors and a relevant question to ask the mailing list is: Where should we put the EXT4 driver?
> Michael said there are other filesystems in MdeModulePkg, but it might be getting too big and proposed the following options:
> 
> 1) EXT4 in new package in edk2 repo as a peer to FatPkg.
> 2) EXT4 in edk2 repo in MdeModulePkg
> 3) EXT4 in edk2-platforms advanced feature package.
> 4) EXT4 in edk2 advanced feature package
> 
> As someone that's still learning how to navigate the project's tree(s), this is a bit over my head and so I'd like your opinion on the matter.
> Also, I would love if someone could point me to some good reading material and/or examples of the package/build system, as I couldn't find documentation on those
> and my previous experiment with Tianocore involved looking at FatPkg and mindlessly copying what it was doing.
> 
> Thanks,
> 
> Pedro
> 
> 
> 
> 
> 
> 

Re: [edk2-devel] GSOC 2021 EXT4 driver Project

[Michael Kubacki]

I had done some of the early work in moving content to 
edk2-platforms/Features. Just wanted to add my support for maintaining 
this as a vendor neutral feature there.

#4 in the options list is not really what AdvancedFeaturePkg intended 
for, there's more details here - 
https://github.com/tianocore/edk2-platforms/tree/master/Features/Intel#AdvancedFeaturePkg.

- Michael

On 6/10/2021 1:54 PM, Leif Lindholm wrote:
> edk2-platforms/Features/Ext4Pkg sounds good to me too.
> 
> /
>      Leif
> 
> On Thu, Jun 10, 2021 at 17:38:17 +0000, Michael D Kinney wrote:
>> Hi Pedro,
>>
>> After thinking about this, I think I would prefer Option (4).
>>
>> The proposed landing zone would be a new Ext4Pkg in the edk2-platforms repository in the Features directory.
>>
>> edk2-platforms/Features/Intel/Ext4Pkg
>>
>> All the features in that directory and not specific to Intel.  There are other email discussions about moving some of that content up a level, so an alternative path would be:
>>
>>                edk2-platforms/Features/Ext4Pkg
>>
>> Best regards,
>>
>> Mike
>>
>>
>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Pedro Falcato
>> Sent: Monday, May 24, 2021 12:27 PM
>> To: devel@edk2.groups.io
>> Subject: [edk2-devel] GSOC 2021 EXT4 driver Project
>>
>> Hi everyone,
>>
>> Me and my project have been selected for GSoC this year, under Michael Kinney and bret. Thank you for the opportunity to collaborate with you and improve Tianocore!
>> If anyone has any questions, please fire away :)
>> How do I get started? I'd like to find some easier tasks as to start trying out patch submission and generally programming in a firmware environment.
>> Also, I've been talking with my mentors and a relevant question to ask the mailing list is: Where should we put the EXT4 driver?
>> Michael said there are other filesystems in MdeModulePkg, but it might be getting too big and proposed the following options:
>>
>> 1) EXT4 in new package in edk2 repo as a peer to FatPkg.
>> 2) EXT4 in edk2 repo in MdeModulePkg
>> 3) EXT4 in edk2-platforms advanced feature package.
>> 4) EXT4 in edk2 advanced feature package
>>
>> As someone that's still learning how to navigate the project's tree(s), this is a bit over my head and so I'd like your opinion on the matter.
>> Also, I would love if someone could point me to some good reading material and/or examples of the package/build system, as I couldn't find documentation on those
>> and my previous experiment with Tianocore involved looking at FatPkg and mindlessly copying what it was doing.
>>
>> Thanks,
>>
>> Pedro
>>
>>
>>
>>
>>
>>
> 
> 
> 
> 

Re: [edk2-devel] GSOC 2021 EXT4 driver Project

[Michael D Kinney]

Hi Michael,

Thanks for the feedback.

#4 should be clarified as adding a new feature package called Ext4Pkg.  Not adding EXT4 to the existing AdvancedFeaturePkg.

Mike 



> -----Original Message-----
> From: Michael Kubacki <mikuback@linux.microsoft.com>
> Sent: Friday, June 11, 2021 11:23 AM
> To: devel@edk2.groups.io; leif@nuviainc.com; Kinney, Michael D <michael.d.kinney@intel.com>
> Cc: pedro.falcato@gmail.com
> Subject: Re: [edk2-devel] GSOC 2021 EXT4 driver Project
> 
> I had done some of the early work in moving content to
> edk2-platforms/Features. Just wanted to add my support for maintaining
> this as a vendor neutral feature there.
> 
> #4 in the options list is not really what AdvancedFeaturePkg intended
> for, there's more details here -
> https://github.com/tianocore/edk2-platforms/tree/master/Features/Intel#AdvancedFeaturePkg.
> 
> - Michael
> 
> On 6/10/2021 1:54 PM, Leif Lindholm wrote:
> > edk2-platforms/Features/Ext4Pkg sounds good to me too.
> >
> > /
> >      Leif
> >
> > On Thu, Jun 10, 2021 at 17:38:17 +0000, Michael D Kinney wrote:
> >> Hi Pedro,
> >>
> >> After thinking about this, I think I would prefer Option (4).
> >>
> >> The proposed landing zone would be a new Ext4Pkg in the edk2-platforms repository in the Features directory.
> >>
> >> edk2-platforms/Features/Intel/Ext4Pkg
> >>
> >> All the features in that directory and not specific to Intel.  There are other email discussions about moving some of
> that content up a level, so an alternative path would be:
> >>
> >>                edk2-platforms/Features/Ext4Pkg
> >>
> >> Best regards,
> >>
> >> Mike
> >>
> >>
> >> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Pedro Falcato
> >> Sent: Monday, May 24, 2021 12:27 PM
> >> To: devel@edk2.groups.io
> >> Subject: [edk2-devel] GSOC 2021 EXT4 driver Project
> >>
> >> Hi everyone,
> >>
> >> Me and my project have been selected for GSoC this year, under Michael Kinney and bret. Thank you for the opportunity
> to collaborate with you and improve Tianocore!
> >> If anyone has any questions, please fire away :)
> >> How do I get started? I'd like to find some easier tasks as to start trying out patch submission and generally
> programming in a firmware environment.
> >> Also, I've been talking with my mentors and a relevant question to ask the mailing list is: Where should we put the
> EXT4 driver?
> >> Michael said there are other filesystems in MdeModulePkg, but it might be getting too big and proposed the following
> options:
> >>
> >> 1) EXT4 in new package in edk2 repo as a peer to FatPkg.
> >> 2) EXT4 in edk2 repo in MdeModulePkg
> >> 3) EXT4 in edk2-platforms advanced feature package.
> >> 4) EXT4 in edk2 advanced feature package
> >>
> >> As someone that's still learning how to navigate the project's tree(s), this is a bit over my head and so I'd like your
> opinion on the matter.
> >> Also, I would love if someone could point me to some good reading material and/or examples of the package/build system,
> as I couldn't find documentation on those
> >> and my previous experiment with Tianocore involved looking at FatPkg and mindlessly copying what it was doing.
> >>
> >> Thanks,
> >>
> >> Pedro
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> > 
> >