From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (NAM12-BN8-obe.outbound.protection.outlook.com [40.107.237.58]) by mx.groups.io with SMTP id smtpd.web12.4099.1589850527452754604 for ; Mon, 18 May 2020 18:08:47 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@vmware.com header.s=selector2 header.b=dZD+ajp+; spf=pass (domain: vmware.com, ip: 40.107.237.58, mailfrom: awarkentin@vmware.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WtP9Ga/6AMLaYWSqBYSr/suOnHp+T7PG4tt3jhnKFyKi+hJbEiP6DjRtJCX//pTdSrazKFVABPx6IRj5ujvgd3/UVUc4uhhuN1yumICfI1T0dqcMv12JA24yt5w1i67svk13IiOkBS74TF6rRaEzEovt5MDOJ6d1J63vFmbPU0VgGa7lYEV5mRE63rFRTbT7A2UK/9L3afWeRHLvPHm1tnVSrUPzbtYwZ0hbe+u7GCVPAY3slkyvTEpr7530N6vPV2DCqpE3qO071QGBr0BmZTGD06xuTfXWTVitl3klX1Qk6sVXGsHPtQwfwY2jve+Fv3vfnm9doSdzHqomfOpfLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=c8cYCMaDtjcRCIw9oJkMHI/Q12STsfk6RwwFdcIPrp4=; b=FFeHvBThZseOFJS3+9hhoybwx/by9yvcudXUX2DLSojdt2vcV/45nVqWb/2kFBUo+4XH9VOjic6DUD+WUum+iLIPqEPKWbX+o7GxaHd+EpG7G8IQP8vYxU1oCUjPp1I720OCcUKXIO79LvpAfCsA5ZckMUI9yd2ECRJ0WnsxbaJzK9JKFpvZg9aPEpSFNa/WszzDLqrUV7zOLIB3YMvzfNnNobuMXhsQISEFwSc3ESChV3prfcU6KDxBcxCtqDNm/2h/zFpyuKPF8a2ja/iucCcA4vqnylYdQyFR/zMvV7SbRf6T/WcpMqyE8me97tnkyQ7uRO4f6OrqQTfsCDmJQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vmware.com; dmarc=pass action=none header.from=vmware.com; dkim=pass header.d=vmware.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmware.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=c8cYCMaDtjcRCIw9oJkMHI/Q12STsfk6RwwFdcIPrp4=; b=dZD+ajp+tK0vEnlfbubK/gCh1Pq2ZP5DhhnVyztosHOCVZobhm2zW7vi/GOftquNekjA2G4l5ztKD6oNHeAuE0LSI+K5EW8tjrPQWciUGGZOlNHGRPWq7jMxEAS9oL+7RXkuUeAYTNK2IBOEflNO1Guq0q2DmbMgg6uoqyAysfk= Received: from BN6PR05MB3411.namprd05.prod.outlook.com (2603:10b6:405:43::23) by BN6PR05MB3457.namprd05.prod.outlook.com (2603:10b6:405:44::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.12; Tue, 19 May 2020 01:08:44 +0000 Received: from BN6PR05MB3411.namprd05.prod.outlook.com ([fe80::f463:db64:43d8:5a0f]) by BN6PR05MB3411.namprd05.prod.outlook.com ([fe80::f463:db64:43d8:5a0f%3]) with mapi id 15.20.3021.019; Tue, 19 May 2020 01:08:44 +0000 From: "Andrei Warkentin" To: "devel@edk2.groups.io" , "andrey.warkentin@gmail.com" CC: "maciej.rabeda@linux.intel.com" , "jiaxin.wu@intel.com" , "siyuan.fu@intel.com" Subject: Re: [edk2-devel] [edk2][PATCH 1/1] HttpBoot: handle servers which may FIN after file sizing in HttpBootLoadFile Thread-Topic: [edk2-devel] [edk2][PATCH 1/1] HttpBoot: handle servers which may FIN after file sizing in HttpBootLoadFile Thread-Index: AQHWKb2ljqhLZ3odGE+9UoW/mm+ucqiuoDnX Date: Tue, 19 May 2020 01:08:44 +0000 Message-ID: References: <20200514070223.48194-1-andrey.warkentin@gmail.com> In-Reply-To: <20200514070223.48194-1-andrey.warkentin@gmail.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: edk2.groups.io; dkim=none (message not signed) header.d=none;edk2.groups.io; dmarc=none action=none header.from=vmware.com; x-originating-ip: [98.214.99.181] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b5255699-eef1-46ea-6569-08d7fb912d53 x-ms-traffictypediagnostic: BN6PR05MB3457: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6108; x-forefront-prvs: 040866B734 x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: wZz3JHqG4/NGYwtsj1apfCbnLwXC9V8UYDRhMv1FXeUlp6zn8dIJUoRg+sZ/143ntmAMZf3RyY8ve34WEH5XZ/CfA9Wky3v+R1LwDG8ZkXDqJvVFtazdBhpE/Nunc+abWBJnatQLzcdtsxqRyKFIICNLCLGzNVNI15YYH6CGnoya8bu8SiYHHvXz2V+NQT+O17VBo4BkVfSI0pIJnITUaz9vfUB6RDARnaSSKRZMekmloUrlNDF/vkygXpy5b4k1pzonxxIcx5BhLnSEb9JzalnGsxmJxMgSHD31OE7gBIGV1i9VtHbp5GoZabM7kve/oNa7fcb0GQCbOzeP9AovFkS/SMhW8ynQtsxeAL33EjxOY8RIKJIBPeMV/I4WNQpdBmtCMqtR5EFA+voVYEH+UA== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BN6PR05MB3411.namprd05.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(4636009)(346002)(376002)(366004)(39860400002)(396003)(136003)(9686003)(66476007)(19627405001)(71200400001)(26005)(166002)(186003)(52536014)(54906003)(55016002)(33656002)(316002)(110136005)(478600001)(966005)(2906002)(4326008)(66946007)(45080400002)(66446008)(64756008)(66556008)(5660300002)(76116006)(7696005)(86362001)(6506007)(8676002)(8936002)(53546011);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata: ZwHu4RbHNgNpt3v7qnF02Q66KM6zINnXM5c/IDiDTQR/vIVP2BRd+VDCLFqad8dUSibiCqDe2ykuV2iD1KFHUYS/inPMrCQsIH0h3yJa1RTu7624j+E/3mHnbTIWZCO9BtqhQzd79Rlpt4uySxQ2eOnYs/HEvGsUgaQtrIdCqVGeKWsJYi4qx6sL6d+Ic5rmn2cwQh39klw6yk0YOXEV+Qe5rvXac5lWVXsifhrw4sYYMX0EIaqFop1iXC5x9Gs2H5jTJ149RKqqUw83igZ69XL/EF90FWTiCJt3/zB3sUh7mFrQ3nC4nzJlsg8QLzbnfY3VribojqVQEghmqjzhYrN6o+6TlsVtng/fTAz6JR0FwsjZIDM8ADAIEvXgTvxpR+e71y1s+iGYC8V2XVNZ28SdDSkrB9V3aW7/vUi5s4RxUYjsXlfF+mmv8JTAdcWSwz09N0jkTWHS2OW/1se5Q2XncXzpTnePcsIGnytBvkQ= x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: vmware.com X-MS-Exchange-CrossTenant-Network-Message-Id: b5255699-eef1-46ea-6569-08d7fb912d53 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 May 2020 01:08:44.3176 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ACOzZhG7AOae/pqepEv9COAzVxNb1O+zDvtUPYnXpPXHdfBg4QgUSbv/iKW/2TKMfr+sTVZuWKX2K2UZJijomA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR05MB3457 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_BN6PR05MB3411AE8548363C6A11AB596BB9B90BN6PR05MB3411namp_" --_000_BN6PR05MB3411AE8548363C6A11AB596BB9B90BN6PR05MB3411namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Anyone? FWIW I've been doing at least 20 boots a day with this, 200-300 MiB at a t= ime. Super stable. This is also tracked now https://bugzilla.tianocore.org/show_bug.cgi?id=3D= 2720 A ________________________________ From: devel@edk2.groups.io on behalf of Andrei Wark= entin via groups.io Sent: Thursday, May 14, 2020 2:02 AM To: devel@edk2.groups.io Cc: maciej.rabeda@linux.intel.com ; jiaxin.= wu@intel.com ; siyuan.fu@intel.com Subject: [edk2-devel] [edk2][PATCH 1/1] HttpBoot: handle servers which may= FIN after file sizing in HttpBootLoadFile Python http.server seems to FIN after the first HEAD request to size the loaded file is completed and ACKed. What happens next is interesting. On low latency connections, the GET request to download may get sent after the server sends the FIN but before the client has a chance to process it. The net result is: - Server ignores GET - HttpBootLoadFile returns EFI_CONNECTION_FIN. Boot fails. In the other case, client handles the FIN before attempting the GET, so there's a proper three-way handshake as part of GET. The solution is to retry HttpBootLoadFile 2 times if it returns EFI_CONNECTION_FIN. This is because HttpBootLoadFile may issue up to 3 requests: HEAD/GET to get size and final GET to load. Some servers may send a FIN after each request. The first request (HEAD) is not supposed to fail this way, so only the two subsequent GET request may result in a FIN. Fixes https://nam04.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2F= bugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D2720&data=3D02%7C01%7Cawar= kentin%40vmware.com%7C73254836459946b2441e08d7f7d4c742%7Cb39138ca3cee4b4aa4= d6cd83d9dd62f0%7C0%7C0%7C637250365557622212&sdata=3DHOKwm430VLz1xB9yyrF= 3T7XKgYipWy5%2FH%2BKbWuIhm24%3D&reserved=3D0 Signed-off-by: Andrei Warkentin --- NetworkPkg/HttpBootDxe/HttpBootImpl.c | 27 +++++++++++++++++++- NetworkPkg/HttpDxe/HttpImpl.c | 5 ++++ 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/NetworkPkg/HttpBootDxe/HttpBootImpl.c b/NetworkPkg/HttpBootDx= e/HttpBootImpl.c index 4a51f35cdd..2d74d5f293 100644 --- a/NetworkPkg/HttpBootDxe/HttpBootImpl.c +++ b/NetworkPkg/HttpBootDxe/HttpBootImpl.c @@ -288,6 +288,7 @@ HttpBootDhcp ( @retval EFI_NOT_STARTED The driver is in stopped state. @retval EFI_BUFFER_TOO_SMALL The BufferSize is too small to read= the boot file. BufferSize has been updated with the size needed t= o complete the request. + @retval EFI_CONNECTION_FIN Server had closed the connection wh= ile we were waiting downloading. @retval EFI_DEVICE_ERROR An unexpected network error occurre= d. @retval Others Other errors as indicated. @@ -535,6 +536,7 @@ HttpBootStop ( @retval EFI_BUFFER_TOO_SMALL The BufferSize is too small to read the c= urrent directory entry. BufferSize has been updated with the size= needed to complete the request. + @retval EFI_CONNECTION_FIN Server had closed the connection while we= were waiting for a response. **/ EFI_STATUS @@ -553,6 +555,7 @@ HttpBootDxeLoadFile ( BOOLEAN UsingIpv6; EFI_STATUS Status; HTTP_BOOT_IMAGE_TYPE ImageType; + UINTN MaxTries; if (This =3D=3D NULL || BufferSize =3D=3D NULL || FilePath =3D=3D NULL)= { return EFI_INVALID_PARAMETER; @@ -598,7 +601,29 @@ HttpBootDxeLoadFile ( // Load the boot file. // ImageType =3D ImageTypeMax; - Status =3D HttpBootLoadFile (Private, BufferSize, Buffer, &ImageType); + // + // HttpBootLoadFile may issue up to 2 requests: HEAD/GET to get + // size and final GET to load. Some servers may send a FIN after + // each request. The first request (HEAD) is not supposed to + // fail this way, so only the two possible GETs need the special + // handling. + // + MaxTries =3D 2; + do { + Status =3D HttpBootLoadFile (Private, BufferSize, Buffer, &ImageType)= ; + if (Status =3D=3D EFI_CONNECTION_FIN) { + if (Private->HttpCreated) { + // + // Tear down HTTP/TCP state entirely. Http->Configure (NULL) is n= ot + // sufficient (EFI_ACCESS_DENIED from TCP stack on subsequent + // HttpBootLoadFile. + // + HttpIoDestroyIo (&Private->HttpIo); + Private->HttpCreated =3D FALSE; + } + } + } while (MaxTries-- && Status =3D=3D EFI_CONNECTION_FIN); + if (EFI_ERROR (Status)) { if (Status =3D=3D EFI_BUFFER_TOO_SMALL && (ImageType =3D=3D ImageType= VirtualCd || ImageType =3D=3D ImageTypeVirtualDisk)) { Status =3D EFI_WARN_FILE_SYSTEM; diff --git a/NetworkPkg/HttpDxe/HttpImpl.c b/NetworkPkg/HttpDxe/HttpImpl.c index 5a6ecbc9d9..34a33b09f7 100644 --- a/NetworkPkg/HttpDxe/HttpImpl.c +++ b/NetworkPkg/HttpDxe/HttpImpl.c @@ -959,6 +959,8 @@ HttpBodyParserCallback ( @retval EFI_OUT_OF_RESOURCES Failed to complete the operation due to= lack of resources. @retval EFI_NOT_READY Can't find a corresponding Tx4Token/Tx6= Token or the EFI_HTTP_UTILITIES_PROTOCOL is not = available. + @retval EFI_CONNECTION_FIN Server had closed the connection while = we were waiting for + a response. **/ EFI_STATUS @@ -1528,6 +1530,9 @@ Error: @retval EFI_OUT_OF_RESOURCES Could not allocate enough system resour= ces. @retval EFI_ACCESS_DENIED An open TCP connection is not present w= ith the host specified by response URL. + + @retval EFI_CONNECTION_FIN Server had closed the connection while = we were waiting for + a response. **/ EFI_STATUS EFIAPI -- 2.17.1 --_000_BN6PR05MB3411AE8548363C6A11AB596BB9B90BN6PR05MB3411namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Anyone?

FWIW I've been doing at least 20 boots a day with this, 200-300 MiB at a t= ime. Super stable.


A
From: devel@edk2.groups.io= <devel@edk2.groups.io> on behalf of Andrei Warkentin via groups.io &= lt;andrey.warkentin=3Dgmail.com@groups.io>
Sent: Thursday, May 14, 2020 2:02 AM
To: devel@edk2.groups.io <devel@edk2.groups.io>
Cc: maciej.rabeda@linux.intel.com <maciej.rabeda@linux.intel.com= >; jiaxin.wu@intel.com <jiaxin.wu@intel.com>; siyuan.fu@intel.com = <siyuan.fu@intel.com>
Subject: [edk2-devel] [edk2][PATCH 1/1] HttpBoot: handle servers wh= ich may FIN after file sizing in HttpBootLoadFile
 
Python http.server seems to FIN after the first H= EAD request to size
the loaded file is completed and ACKed. What happens next is interesting.<= br> On low latency connections, the GET request to download may get sent
after the server sends the FIN but before the client has a chance to
process it. The net result is:
- Server ignores GET
- HttpBootLoadFile returns EFI_CONNECTION_FIN. Boot fails.

In the other case, client handles the FIN before attempting the GET,
so there's a proper three-way handshake as part of GET.

The solution is to retry HttpBootLoadFile 2 times if it returns
EFI_CONNECTION_FIN. This is because HttpBootLoadFile may issue up to
3 requests: HEAD/GET to get size and final GET to load. Some servers
may send a FIN after each request. The first request (HEAD) is not
supposed to fail this way, so only the two subsequent GET request
may result in a FIN.

Fixes https://nam04.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fbugzil= la.tianocore.org%2Fshow_bug.cgi%3Fid%3D2720&amp;data=3D02%7C01%7Cawarke= ntin%40vmware.com%7C73254836459946b2441e08d7f7d4c742%7Cb39138ca3cee4b4aa4d6= cd83d9dd62f0%7C0%7C0%7C637250365557622212&amp;sdata=3DHOKwm430VLz1xB9yy= rF3T7XKgYipWy5%2FH%2BKbWuIhm24%3D&amp;reserved=3D0

Signed-off-by: Andrei Warkentin <andrey.warkentin@gmail.com>
---
 NetworkPkg/HttpBootDxe/HttpBootImpl.c | 27 +++++= ++++++++++++++-
 NetworkPkg/HttpDxe/HttpImpl.c      &nb= sp;  |  5 ++++
 2 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/NetworkPkg/HttpBootDxe/HttpBootImpl.c b/NetworkPkg/HttpBootDx= e/HttpBootImpl.c
index 4a51f35cdd..2d74d5f293 100644
--- a/NetworkPkg/HttpBootDxe/HttpBootImpl.c
+++ b/NetworkPkg/HttpBootDxe/HttpBootImpl.c
@@ -288,6 +288,7 @@ HttpBootDhcp (
   @retval EFI_NOT_STARTED      &n= bsp;      The driver is in stopped state.
   @retval EFI_BUFFER_TOO_SMALL     &nb= sp;  The BufferSize is too small to read the boot file. BufferSize has=
            &n= bsp;            = ;            &n= bsp; been updated with the size needed to complete the request.
+  @retval EFI_CONNECTION_FIN      =     Server had closed the connection while we were waiting d= ownloading.
   @retval EFI_DEVICE_ERROR      &= nbsp;     An unexpected network error occurred.
   @retval Others        = ;            &n= bsp; Other errors as indicated.
 
@@ -535,6 +536,7 @@ HttpBootStop (
   @retval EFI_BUFFER_TOO_SMALL  The BufferSize is too smal= l to read the current directory entry.
            &n= bsp;            = ;        BufferSize has been updated wit= h the size needed to complete
            &n= bsp;            = ;        the request.
+  @retval EFI_CONNECTION_FIN    Server had closed= the connection while we were waiting for a response.
 
 **/
 EFI_STATUS
@@ -553,6 +555,7 @@ HttpBootDxeLoadFile (
   BOOLEAN         =             &nb= sp; UsingIpv6;
   EFI_STATUS        &nb= sp;           Status;
   HTTP_BOOT_IMAGE_TYPE       = ;   ImageType;
+  UINTN         &nb= sp;            =    MaxTries;
 
   if (This =3D=3D NULL || BufferSize =3D=3D NULL || FilePath = =3D=3D NULL) {
     return EFI_INVALID_PARAMETER;
@@ -598,7 +601,29 @@ HttpBootDxeLoadFile (
   // Load the boot file.
   //
   ImageType =3D ImageTypeMax;
-  Status =3D HttpBootLoadFile (Private, BufferSize, Buffer, &Ima= geType);
+  //
+  // HttpBootLoadFile may issue up to 2 requests: HEAD/GET to ge= t
+  // size and final GET to load. Some servers may send a FIN aft= er
+  // each request. The first request (HEAD) is not supposed to +  // fail this way, so only the two possible GETs need the speci= al
+  // handling.
+  //
+  MaxTries =3D 2;
+  do {
+    Status =3D HttpBootLoadFile (Private, BufferSize, = Buffer, &ImageType);
+    if (Status =3D=3D EFI_CONNECTION_FIN) {
+      if (Private->HttpCreated) {
+        //
+        // Tear down HTTP/TCP stat= e entirely. Http->Configure (NULL) is not
+        // sufficient (EFI_ACCESS_= DENIED from TCP stack on subsequent
+        // HttpBootLoadFile.
+        //
+        HttpIoDestroyIo (&Priv= ate->HttpIo);
+        Private->HttpCreated = =3D FALSE;
+      }
+    }
+  } while (MaxTries-- && Status =3D=3D EFI_CONNECTION_FI= N);
+
   if (EFI_ERROR (Status)) {
     if (Status =3D=3D EFI_BUFFER_TOO_SMALL &&= (ImageType =3D=3D ImageTypeVirtualCd || ImageType =3D=3D ImageTypeVirtualD= isk)) {
       Status =3D EFI_WARN_FILE_SYSTEM;
diff --git a/NetworkPkg/HttpDxe/HttpImpl.c b/NetworkPkg/HttpDxe/HttpImpl.c=
index 5a6ecbc9d9..34a33b09f7 100644
--- a/NetworkPkg/HttpDxe/HttpImpl.c
+++ b/NetworkPkg/HttpDxe/HttpImpl.c
@@ -959,6 +959,8 @@ HttpBodyParserCallback (
   @retval EFI_OUT_OF_RESOURCES    Failed to comp= lete the operation due to lack of resources.
   @retval EFI_NOT_READY      &nbs= p;    Can't find a corresponding Tx4Token/Tx6Token or
            &n= bsp;            = ;          the EFI_HTTP_UTILIT= IES_PROTOCOL is not available.
+  @retval EFI_CONNECTION_FIN      Serve= r had closed the connection while we were waiting for
+           &nb= sp;            =           a response.
 
 **/
 EFI_STATUS
@@ -1528,6 +1530,9 @@ Error:
   @retval EFI_OUT_OF_RESOURCES    Could not allo= cate enough system resources.
   @retval EFI_ACCESS_DENIED      = An open TCP connection is not present with the host
            &n= bsp;            = ;          specified by respon= se URL.
+
+  @retval EFI_CONNECTION_FIN      Serve= r had closed the connection while we were waiting for
+           &nb= sp;            =           a response.
 **/
 EFI_STATUS
 EFIAPI
--
2.17.1




--_000_BN6PR05MB3411AE8548363C6A11AB596BB9B90BN6PR05MB3411namp_--