Http redirection handling using HttpDxe driver

Http redirection handling using HttpDxe driver

[Vladimir Olovyannikov]

Hi Maciej,

I would like to get your input (and the HttpDxe driver guys) on the issue
I see with Http redirection (using HttpDynamicCommand)
Basically, let's say that we connect to a server "server.com"

Request:
GET / HTTP/1.1
Host: server.com

Reply:
HTTP/1.1 301 Moved permanently
Location: www.server.com

By RFC, the client reissues the request like following and connects to
www.server.com:
Request:
GET / HTTP/1.1
Host: www.server.com

Reply:
HTTP/1.1 200 OK

I have an issue of getting EFI_ACCESS_DENIED in SockInterface.c on the
redirection.
The logic is:
1. Send a request
2. Parse the response. If there is a redirection code returned, then
 a) cancel Http connection
 b) resubmit to the proper host with proper host header.

Obviously, there are no issues with a). However, there is an issue with b)
When the second request is being sent,
EfiHttpRequest sets Configure and Reconfigure flags,  closes and cancels
Http connection
with HttpCloseConnection/EfiHttpCancel.
Everything is OK at this point. Note that the socket is closed, but it's
ConfiguredState is 1, and is not changed.
Now, HttpInitSession() is called with Configure flag.
  It calls into HttpConfigureTcp4(), which in turn calls
    Tcp4Configure().
    Here everything is fine until
      SockConfigure is called.
      The SockConfigure checks if ConfigureState is "not configured" (see
above, the ConfigureState is not changed on Http session cancel/close).
       So, the SockConfigure returns EFI_ACCESS_DENIED error as the socket
is configured, and this is returned to HttpConfigureTcp4().
  Which, in turn, returns an error up, and the Http in HttpDynamicCommand
gets this error, reports it and aborts the operation.

If I change the line in HttpConfigureTcp4, line 1108 in HttpProto.c from
if (EFI_ERROR (Status)) {
  DEBUG ((.....));
  return Status;
}
to
if (EFI_ERROR (Status) && (Status != EFI_ACCESS_DENIED)) {
  DEBUG ((...));
  return Status;
}
then EFI_SUCCESS is returned, socket placeholder is reused, and multiple
redirections works properly (I tried this scheme:
server.com->www.server.com->www1.server.com->got content).
I suspect, it is wrong to make an assumption that the ACCESS_DEINED
returned from Tcp4Configure() would be because of the socket.
Maybe, I am missing something? Can you help?

Thank you,
Vladimir

Re: Http redirection handling using HttpDxe driver

[Maciej Rabeda]

Hi Vladimir,

I got your message, I will look into it next week.
Have a great weekend!

Maciej

On 27-Aug-20 21:23, Vladimir Olovyannikov wrote:
> Hi Maciej,
>
> I would like to get your input (and the HttpDxe driver guys) on the issue
> I see with Http redirection (using HttpDynamicCommand)
> Basically, let's say that we connect to a server "server.com"
>
> Request:
> GET / HTTP/1.1
> Host: server.com
>
> Reply:
> HTTP/1.1 301 Moved permanently
> Location: www.server.com
>
> By RFC, the client reissues the request like following and connects to
> www.server.com:
> Request:
> GET / HTTP/1.1
> Host: www.server.com
>
> Reply:
> HTTP/1.1 200 OK
>
> I have an issue of getting EFI_ACCESS_DENIED in SockInterface.c on the
> redirection.
> The logic is:
> 1. Send a request
> 2. Parse the response. If there is a redirection code returned, then
>   a) cancel Http connection
>   b) resubmit to the proper host with proper host header.
>
> Obviously, there are no issues with a). However, there is an issue with b)
> When the second request is being sent,
> EfiHttpRequest sets Configure and Reconfigure flags,  closes and cancels
> Http connection
> with HttpCloseConnection/EfiHttpCancel.
> Everything is OK at this point. Note that the socket is closed, but it's
> ConfiguredState is 1, and is not changed.
> Now, HttpInitSession() is called with Configure flag.
>    It calls into HttpConfigureTcp4(), which in turn calls
>      Tcp4Configure().
>      Here everything is fine until
>        SockConfigure is called.
>        The SockConfigure checks if ConfigureState is "not configured" (see
> above, the ConfigureState is not changed on Http session cancel/close).
>         So, the SockConfigure returns EFI_ACCESS_DENIED error as the socket
> is configured, and this is returned to HttpConfigureTcp4().
>    Which, in turn, returns an error up, and the Http in HttpDynamicCommand
> gets this error, reports it and aborts the operation.
>
> If I change the line in HttpConfigureTcp4, line 1108 in HttpProto.c from
> if (EFI_ERROR (Status)) {
>    DEBUG ((.....));
>    return Status;
> }
> to
> if (EFI_ERROR (Status) && (Status != EFI_ACCESS_DENIED)) {
>    DEBUG ((...));
>    return Status;
> }
> then EFI_SUCCESS is returned, socket placeholder is reused, and multiple
> redirections works properly (I tried this scheme:
> server.com->www.server.com->www1.server.com->got content).
> I suspect, it is wrong to make an assumption that the ACCESS_DEINED
> returned from Tcp4Configure() would be because of the socket.
> Maybe, I am missing something? Can you help?
>
> Thank you,
> Vladimir

Re: Http redirection handling using HttpDxe driver

[Vladimir Olovyannikov]

> -----Original Message-----
> From: Rabeda, Maciej <maciej.rabeda@linux.intel.com>
> Sent: Friday, August 28, 2020 8:27 AM
> To: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com>
> Cc: devel@edk2.groups.io
> Subject: Re: Http redirection handling using HttpDxe driver
>
> Hi Vladimir,
>
> I got your message, I will look into it next week.
> Have a great weekend!
>
Thank you Maciej,
I looked through the code and found that my redirection assumption was
incorrect.
Apparently, one must destroy the socket before resubmitting a request for a
different server.
I am going to submit patchset v8. Please review it when you have a chance.
@Laszlo
This does not affect a direct download when there is no redirection, which
you tested.

Have a great weekend!

Thank you,
Vladimir
> Maciej
>
> On 27-Aug-20 21:23, Vladimir Olovyannikov wrote:
> > Hi Maciej,
> >
> > I would like to get your input (and the HttpDxe driver guys) on the
> > issue I see with Http redirection (using HttpDynamicCommand)
> > Basically, let's say that we connect to a server "server.com"
> >
> > Request:
> > GET / HTTP/1.1
> > Host: server.com
> >
> > Reply:
> > HTTP/1.1 301 Moved permanently
> > Location: www.server.com
> >
> > By RFC, the client reissues the request like following and connects to
> > www.server.com:
> > Request:
> > GET / HTTP/1.1
> > Host: www.server.com
> >
> > Reply:
> > HTTP/1.1 200 OK
> >
> > I have an issue of getting EFI_ACCESS_DENIED in SockInterface.c on the
> > redirection.
> > The logic is:
> > 1. Send a request
> > 2. Parse the response. If there is a redirection code returned, then
> >   a) cancel Http connection
> >   b) resubmit to the proper host with proper host header.
> >
> > Obviously, there are no issues with a). However, there is an issue
> > with b) When the second request is being sent, EfiHttpRequest sets
> > Configure and Reconfigure flags,  closes and cancels Http connection
> > with HttpCloseConnection/EfiHttpCancel.
> > Everything is OK at this point. Note that the socket is closed, but
> > it's ConfiguredState is 1, and is not changed.
> > Now, HttpInitSession() is called with Configure flag.
> >    It calls into HttpConfigureTcp4(), which in turn calls
> >      Tcp4Configure().
> >      Here everything is fine until
> >        SockConfigure is called.
> >        The SockConfigure checks if ConfigureState is "not configured"
> > (see above, the ConfigureState is not changed on Http session
> cancel/close).
> >         So, the SockConfigure returns EFI_ACCESS_DENIED error as the
> > socket is configured, and this is returned to HttpConfigureTcp4().
> >    Which, in turn, returns an error up, and the Http in
> > HttpDynamicCommand gets this error, reports it and aborts the operation.
> >
> > If I change the line in HttpConfigureTcp4, line 1108 in HttpProto.c
> > from if (EFI_ERROR (Status)) {
> >    DEBUG ((.....));
> >    return Status;
> > }
> > to
> > if (EFI_ERROR (Status) && (Status != EFI_ACCESS_DENIED)) {
> >    DEBUG ((...));
> >    return Status;
> > }
> > then EFI_SUCCESS is returned, socket placeholder is reused, and
> > multiple redirections works properly (I tried this scheme:
> > server.com->www.server.com->www1.server.com->got content).
> > I suspect, it is wrong to make an assumption that the ACCESS_DEINED
> > returned from Tcp4Configure() would be because of the socket.
> > Maybe, I am missing something? Can you help?
> >
> > Thank you,
> > Vladimir

Re: [edk2-devel] Http redirection handling using HttpDxe driver

[Sean]

I think this is the similar to issue 1674 which was reported in early 2019.
https://bugzilla.tianocore.org/show_bug.cgi?id=1674

I'll check internally to find out what we ended up doing to work around 
it.  I think we closed the socket and opened a new one.  This was less 
than ideal and is not intuitive given the API.

Thanks
Sean






On 8/28/2020 10:04 AM, Vladimir Olovyannikov via groups.io wrote:
>> -----Original Message-----
>> From: Rabeda, Maciej <maciej.rabeda@linux.intel.com>
>> Sent: Friday, August 28, 2020 8:27 AM
>> To: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com>
>> Cc: devel@edk2.groups.io
>> Subject: Re: Http redirection handling using HttpDxe driver
>>
>> Hi Vladimir,
>>
>> I got your message, I will look into it next week.
>> Have a great weekend!
>>
> Thank you Maciej,
> I looked through the code and found that my redirection assumption was
> incorrect.
> Apparently, one must destroy the socket before resubmitting a request for a
> different server.
> I am going to submit patchset v8. Please review it when you have a chance.
> @Laszlo
> This does not affect a direct download when there is no redirection, which
> you tested.
> 
> Have a great weekend!
> 
> Thank you,
> Vladimir
>> Maciej
>>
>> On 27-Aug-20 21:23, Vladimir Olovyannikov wrote:
>>> Hi Maciej,
>>>
>>> I would like to get your input (and the HttpDxe driver guys) on the
>>> issue I see with Http redirection (using HttpDynamicCommand)
>>> Basically, let's say that we connect to a server "server.com"
>>>
>>> Request:
>>> GET / HTTP/1.1
>>> Host: server.com
>>>
>>> Reply:
>>> HTTP/1.1 301 Moved permanently
>>> Location: www.server.com
>>>
>>> By RFC, the client reissues the request like following and connects to
>>> www.server.com:
>>> Request:
>>> GET / HTTP/1.1
>>> Host: www.server.com
>>>
>>> Reply:
>>> HTTP/1.1 200 OK
>>>
>>> I have an issue of getting EFI_ACCESS_DENIED in SockInterface.c on the
>>> redirection.
>>> The logic is:
>>> 1. Send a request
>>> 2. Parse the response. If there is a redirection code returned, then
>>>    a) cancel Http connection
>>>    b) resubmit to the proper host with proper host header.
>>>
>>> Obviously, there are no issues with a). However, there is an issue
>>> with b) When the second request is being sent, EfiHttpRequest sets
>>> Configure and Reconfigure flags,  closes and cancels Http connection
>>> with HttpCloseConnection/EfiHttpCancel.
>>> Everything is OK at this point. Note that the socket is closed, but
>>> it's ConfiguredState is 1, and is not changed.
>>> Now, HttpInitSession() is called with Configure flag.
>>>     It calls into HttpConfigureTcp4(), which in turn calls
>>>       Tcp4Configure().
>>>       Here everything is fine until
>>>         SockConfigure is called.
>>>         The SockConfigure checks if ConfigureState is "not configured"
>>> (see above, the ConfigureState is not changed on Http session
>> cancel/close).
>>>          So, the SockConfigure returns EFI_ACCESS_DENIED error as the
>>> socket is configured, and this is returned to HttpConfigureTcp4().
>>>     Which, in turn, returns an error up, and the Http in
>>> HttpDynamicCommand gets this error, reports it and aborts the operation.
>>>
>>> If I change the line in HttpConfigureTcp4, line 1108 in HttpProto.c
>>> from if (EFI_ERROR (Status)) {
>>>     DEBUG ((.....));
>>>     return Status;
>>> }
>>> to
>>> if (EFI_ERROR (Status) && (Status != EFI_ACCESS_DENIED)) {
>>>     DEBUG ((...));
>>>     return Status;
>>> }
>>> then EFI_SUCCESS is returned, socket placeholder is reused, and
>>> multiple redirections works properly (I tried this scheme:
>>> server.com->www.server.com->www1.server.com->got content).
>>> I suspect, it is wrong to make an assumption that the ACCESS_DEINED
>>> returned from Tcp4Configure() would be because of the socket.
>>> Maybe, I am missing something? Can you help?
>>>
>>> Thank you,
>>> Vladimir
> 
> 
> 

Re: [edk2-devel] Http redirection handling using HttpDxe driver

[Vladimir Olovyannikov]

> -----Original Message-----
> From: Sean Brogan <spbrogan@outlook.com>
> Sent: Friday, August 28, 2020 8:40 PM
> To: devel@edk2.groups.io; vladimir.olovyannikov@broadcom.com; Rabeda,
> Maciej <maciej.rabeda@linux.intel.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Subject: Re: [edk2-devel] Http redirection handling using HttpDxe driver
>
> I think this is the similar to issue 1674 which was reported in early
> 2019.
> https://bugzilla.tianocore.org/show_bug.cgi?id=1674
Thanks Sean,
Indeed, it looks very similar.
For now, I ended up destroying the child explicitly and recreating it if
reconfigure is needed (redirection), inside my app.
This does not affect the common code and works perfectly for the app.

Thank you,
Vladimir
>
> I'll check internally to find out what we ended up doing to work around
> it.  I
> think we closed the socket and opened a new one.  This was less than ideal
> and is not intuitive given the API.
>
> Thanks
> Sean
>
>
>
>
>
>
> On 8/28/2020 10:04 AM, Vladimir Olovyannikov via groups.io wrote:
> >> -----Original Message-----
> >> From: Rabeda, Maciej <maciej.rabeda@linux.intel.com>
> >> Sent: Friday, August 28, 2020 8:27 AM
> >> To: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com>
> >> Cc: devel@edk2.groups.io
> >> Subject: Re: Http redirection handling using HttpDxe driver
> >>
> >> Hi Vladimir,
> >>
> >> I got your message, I will look into it next week.
> >> Have a great weekend!
> >>
> > Thank you Maciej,
> > I looked through the code and found that my redirection assumption was
> > incorrect.
> > Apparently, one must destroy the socket before resubmitting a request
> > for a different server.
> > I am going to submit patchset v8. Please review it when you have a
> > chance.
> > @Laszlo
> > This does not affect a direct download when there is no redirection,
> > which you tested.
> >
> > Have a great weekend!
> >
> > Thank you,
> > Vladimir
> >> Maciej
> >>
> >> On 27-Aug-20 21:23, Vladimir Olovyannikov wrote:
> >>> Hi Maciej,
> >>>
> >>> I would like to get your input (and the HttpDxe driver guys) on the
> >>> issue I see with Http redirection (using HttpDynamicCommand)
> >>> Basically, let's say that we connect to a server "server.com"
> >>>
> >>> Request:
> >>> GET / HTTP/1.1
> >>> Host: server.com
> >>>
> >>> Reply:
> >>> HTTP/1.1 301 Moved permanently
> >>> Location: www.server.com
> >>>
> >>> By RFC, the client reissues the request like following and connects
> >>> to
> >>> www.server.com:
> >>> Request:
> >>> GET / HTTP/1.1
> >>> Host: www.server.com
> >>>
> >>> Reply:
> >>> HTTP/1.1 200 OK
> >>>
> >>> I have an issue of getting EFI_ACCESS_DENIED in SockInterface.c on
> >>> the redirection.
> >>> The logic is:
> >>> 1. Send a request
> >>> 2. Parse the response. If there is a redirection code returned, then
> >>>    a) cancel Http connection
> >>>    b) resubmit to the proper host with proper host header.
> >>>
> >>> Obviously, there are no issues with a). However, there is an issue
> >>> with b) When the second request is being sent, EfiHttpRequest sets
> >>> Configure and Reconfigure flags,  closes and cancels Http connection
> >>> with HttpCloseConnection/EfiHttpCancel.
> >>> Everything is OK at this point. Note that the socket is closed, but
> >>> it's ConfiguredState is 1, and is not changed.
> >>> Now, HttpInitSession() is called with Configure flag.
> >>>     It calls into HttpConfigureTcp4(), which in turn calls
> >>>       Tcp4Configure().
> >>>       Here everything is fine until
> >>>         SockConfigure is called.
> >>>         The SockConfigure checks if ConfigureState is "not configured"
> >>> (see above, the ConfigureState is not changed on Http session
> >> cancel/close).
> >>>          So, the SockConfigure returns EFI_ACCESS_DENIED error as
> >>> the socket is configured, and this is returned to HttpConfigureTcp4().
> >>>     Which, in turn, returns an error up, and the Http in
> >>> HttpDynamicCommand gets this error, reports it and aborts the
> operation.
> >>>
> >>> If I change the line in HttpConfigureTcp4, line 1108 in HttpProto.c
> >>> from if (EFI_ERROR (Status)) {
> >>>     DEBUG ((.....));
> >>>     return Status;
> >>> }
> >>> to
> >>> if (EFI_ERROR (Status) && (Status != EFI_ACCESS_DENIED)) {
> >>>     DEBUG ((...));
> >>>     return Status;
> >>> }
> >>> then EFI_SUCCESS is returned, socket placeholder is reused, and
> >>> multiple redirections works properly (I tried this scheme:
> >>> server.com->www.server.com->www1.server.com->got content).
> >>> I suspect, it is wrong to make an assumption that the ACCESS_DEINED
> >>> returned from Tcp4Configure() would be because of the socket.
> >>> Maybe, I am missing something? Can you help?
> >>>
> >>> Thank you,
> >>> Vladimir
> >
> > 
> >

Re: [edk2-devel] Http redirection handling using HttpDxe driver

[Michael Turner]

[-- Attachment #1: Type: text/plain, Size: 78 bytes --]

The workaround we used was to open and close a new connection for every URL.

[-- Attachment #2: Type: text/html, Size: 78 bytes --]