Information about the VMs used to do EDK2 CI?

Information about the VMs used to do EDK2 CI?

[Rebecca Cran]

I was wondering if it might be possible to get information about the VMs
used to run EDK2 CI?

For example I noticed the Edk2ToolsBuild.py script doesn't try and build
BaseTools in parallel, causing it to take 45 seconds - whereas it takes
around 10 when run with "make -j4" and 6 with "make -j16". However,
without knowing the specifications of the VMs used, I'm not sure if I
should work on optimizing things like that to run better.


-- 
Rebecca Cran

Re: [EXTERNAL] [edk2-devel] Information about the VMs used to do EDK2 CI?

[Bret Barkelew]

[-- Attachment #1: Type: text/plain, Size: 1578 bytes --]

CI runs on the DevOps pool for ubuntu-latest and windows-latest, respectively…
https://github.com/tianocore/edk2/blob/0c8ea9fe1adbbee230ee0c68f28b68ca2b0534bc/.azurepipelines/Ubuntu-GCC5.yml#L16
https://github.com/tianocore/edk2/blob/0c8ea9fe1adbbee230ee0c68f28b68ca2b0534bc/.azurepipelines/Windows-VS2019.yml#L17

As for OS, it looks like this table is kept up-to-date:
https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops#use-a-microsoft-hosted-agent

I *think* they may be assumed to be single-threaded and parallelism is accomplished via “jobs”:
https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops#capabilities-and-limitations
(@Sean<mailto:sean.brogan@microsoft.com> may correct me on that)

What details do you need and I’ll try to find them. 😊

- Bret

From: Rebecca Cran via Groups.Io<mailto:rebecca=bsdio.com@groups.io>
Sent: Friday, March 20, 2020 6:33 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Subject: [EXTERNAL] [edk2-devel] Information about the VMs used to do EDK2 CI?

I was wondering if it might be possible to get information about the VMs
used to run EDK2 CI?

For example I noticed the Edk2ToolsBuild.py script doesn't try and build
BaseTools in parallel, causing it to take 45 seconds - whereas it takes
around 10 when run with "make -j4" and 6 with "make -j16". However,
without knowing the specifications of the VMs used, I'm not sure if I
should work on optimizing things like that to run better.


--
Rebecca Cran






[-- Attachment #2: Type: text/html, Size: 4528 bytes --]

Re: [EXTERNAL] [edk2-devel] Information about the VMs used to do EDK2 CI?

[Rebecca Cran]

[-- Attachment #1: Type: text/plain, Size: 2038 bytes --]

Thanks! It was the singe-threaded part that I was interested in. But
also, it sounds like for more complex/in-depth testing such as running
QEMU to do runtime validation and regression-testing of OVMF etc. we'll
want to set up our own agents, since the hosted ones are relatively
restrictive.

-- 
Rebecca Cran

On 2020-03-20 19:46, Bret Barkelew wrote:
>
> CI runs on the DevOps pool for ubuntu-latest and windows-latest,
> respectively…
>
> https://github.com/tianocore/edk2/blob/0c8ea9fe1adbbee230ee0c68f28b68ca2b0534bc/.azurepipelines/Ubuntu-GCC5.yml#L16
>
> https://github.com/tianocore/edk2/blob/0c8ea9fe1adbbee230ee0c68f28b68ca2b0534bc/.azurepipelines/Windows-VS2019.yml#L17
>
>  
>
> As for OS, it looks like this table is kept up-to-date:
>
> https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops#use-a-microsoft-hosted-agent
>
>  
>
> I **think** they may be assumed to be single-threaded and parallelism
> is accomplished via “jobs”:
>
> https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops#capabilities-and-limitations
>
> (@Sean <mailto:sean.brogan@microsoft.com> may correct me on that)
>
>  
>
> What details do you need and I’ll try to find them. 😊
>
>  
>
> - Bret
>
>  
>
> *From: *Rebecca Cran via Groups.Io <mailto:rebecca=bsdio.com@groups.io>
> *Sent: *Friday, March 20, 2020 6:33 PM
> *To: *devel@edk2.groups.io <mailto:devel@edk2.groups.io>
> *Subject: *[EXTERNAL] [edk2-devel] Information about the VMs used to
> do EDK2 CI?
>
>  
>
> I was wondering if it might be possible to get information about the VMs
> used to run EDK2 CI?
>
> For example I noticed the Edk2ToolsBuild.py script doesn't try and build
> BaseTools in parallel, causing it to take 45 seconds - whereas it takes
> around 10 when run with "make -j4" and 6 with "make -j16". However,
> without knowing the specifications of the VMs used, I'm not sure if I
> should work on optimizing things like that to run better.
>
>
> -- 
> Rebecca Cran
>
>
>
> 
>
>  
>


[-- Attachment #2: Type: text/html, Size: 5932 bytes --]

Re: [edk2-devel] [EXTERNAL] [edk2-devel] Information about the VMs used to do EDK2 CI?

[Sean]

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

Rebecca - I think for any platform testing it would make more sense to use self hosted agents.  They could even be your own VM.  QEMU and emulator might be the two where we could explore using the DevOps VMs but I worry we will spend too much time downloading and install dependencies since the VM is clean on each run.

We can detect the number of threads available and call make with the associated parameters.  That might help speed it up and would be safe to use.

The doc says they are DS2v2 instances which have the following.

TABLE 2 Size vCPU Memory: GiB Temp storage (SSD) GiB Max data disks Max cached and temp storage throughput: IOPS/MBps (cache size in GiB) Max uncached disk throughput: IOPS/MBps Max NICs/Expected network bandwidth (Mbps) Standard_DS2_v2 2 7 14 8 8000/64 (86) 6400/96 2/1500

[-- Attachment #2: Type: text/html, Size: 6190 bytes --]

Re: [edk2-devel] [EXTERNAL] [edk2-devel] Information about the VMs used to do EDK2 CI?

[Rebecca Cran]

[-- Attachment #1: Type: text/plain, Size: 1170 bytes --]

Thank you. I think we probably do want to detect the number of threads 
available: I'll see if I can work on that.

I'll plan to set up my own Azure self-hosted agent for doing more 
extensive testing for FreeBSD and Bhyve.


-- 
Rebecca Cran


On 3/21/20 10:41 AM, Sean via Groups.Io wrote:
> Rebecca - I think for any platform testing it would make more sense to 
> use self hosted agents.  They could even be your own VM. QEMU and 
> emulator might be the two where we could explore using the DevOps VMs 
> but I worry we will spend too much time downloading and install 
> dependencies since the VM is clean on each run.
>
> We can detect the number of threads available and call make with the 
> associated parameters.  That might help speed it up and would be safe 
> to use.
>
> The doc says they are DS2v2 instances which have the following.
>
> TABLE 2 Size 	vCPU 	Memory: GiB 	Temp storage (SSD) GiB 	Max data 
> disks 	Max cached and temp storage throughput: IOPS/MBps (cache size 
> in GiB) 	Max uncached disk throughput: IOPS/MBps 	Max NICs/Expected 
> network bandwidth (Mbps)
> 							
> Standard_DS2_v2 	2 	7 	14 	8 	8000/64 (86) 	6400/96 	2/1500
>
> 

[-- Attachment #2: Type: text/html, Size: 8418 bytes --]

Re: [edk2-devel] [EXTERNAL] [edk2-devel] Information about the VMs used to do EDK2 CI?

[Laszlo Ersek]

On 03/22/20 00:03, Rebecca Cran wrote:
> Thank you. I think we probably do want to detect the number of threads
> available: I'll see if I can work on that.

In a Linux environment, you can run "getconf _NPROCESSORS_ONLN".

In fact that's how I build the native BaseTools on my laptop:

$ . edksetup.sh
$ nice make -C "$EDK_TOOLS_PATH" -j $(getconf _NPROCESSORS_ONLN)

And from a cursory web search:

  https://stackoverflow.com/a/23569003

it is supposed to work on macOS too, and with a small tweak, on the BSDs
as well.

Thanks
Laszlo

Re: [edk2-devel] [EXTERNAL] [edk2-devel] Information about the VMs used to do EDK2 CI?

[Sean]

[-- Attachment #1: Type: text/plain, Size: 254 bytes --]

Thanks

I opened a bug to get this tracked. https://bugzilla.tianocore.org/show_bug.cgi?id=2640
In that bug i link to a commit which does the above using python.  It saves about 20 seconds for each build on linux in Azure pipelines.

Thanks
Sean

[-- Attachment #2: Type: text/html, Size: 365 bytes --]

Re: [edk2-devel] [EXTERNAL] [edk2-devel] Information about the VMs used to do EDK2 CI?

[Andrew Fish]

[-- Attachment #1: Type: text/plain, Size: 878 bytes --]

This works on my MacBook Pro
$ getconf _NPROCESSORS_ONLN
8

The macOS centric way is this:
$ sysctl -n hw.logicalcpu
8

Thanks,

Andrew FIsh

> On Mar 23, 2020, at 5:38 AM, Laszlo Ersek <lersek@redhat.com> wrote:
> 
> On 03/22/20 00:03, Rebecca Cran wrote:
>> Thank you. I think we probably do want to detect the number of threads
>> available: I'll see if I can work on that.
> 
> In a Linux environment, you can run "getconf _NPROCESSORS_ONLN".
> 
> In fact that's how I build the native BaseTools on my laptop:
> 
> $ . edksetup.sh
> $ nice make -C "$EDK_TOOLS_PATH" -j $(getconf _NPROCESSORS_ONLN)
> 
> And from a cursory web search:
> 
>  https://stackoverflow.com/a/23569003 <https://stackoverflow.com/a/23569003>
> 
> it is supposed to work on macOS too, and with a small tweak, on the BSDs
> as well.
> 
> Thanks
> Laszlo
> 
> 
> 


[-- Attachment #2: Type: text/html, Size: 14789 bytes --]