On Wed, 2019-10-16 at 05:18 +0000, Wu, Jiaxin wrote:
> Fortunately, I get my wanted answer in RFC6125, SECTION 3.1 :
> 
>    If a subjectAltName extension of type dNSName is present, that MUST
>    be used as the identity.  Otherwise, the (most specific) Common Name
>    field in the Subject field of the certificate MUST be used.  Although
>    the use of the Common Name is existing practice, it is deprecated and
>    Certification Authorities are encouraged to use the dNSName instead.
>    ...
>    In some cases, the URI is specified as an IP address rather than a
>    Hostname .  In this case, the iPAddress subjectAltName must be present
>    in the certificate and must exactly match the IP in the URI.
> 
> So, the question is more clear here:
> 1) IP in the URI is used for the iPAddress of SAN verification, but *not say no* if upper driver set to dNSName of SAN.
> 2) iPAddress of SAN MUST be present in the certificate and must exactly match the IP in the URI. If not, we can treat as failure.

Good find. 

RFC5280 §4.2.1.6 is a bit poorly drafted here; I've filed an erratum.

It's very prescriptive about email addresses and DNS names, clearly
starting that those MUST be stored in rfc822Name or dNSName form
respectively:

   When the subjectAltName extension contains an Internet mail address,
   the address MUST be stored in the rfc822Name. 
...
   When the subjectAltName extension contains a domain name system
   label, the domain name MUST be stored in the dNSName…

But it seems less clear about IP addresses, merely saying that *if* you
choose to use the iPAddress form, it must be big-endian:

   When the subjectAltName extension contains an iPAddress, the address
   MUST be stored in the octet string in "network byte order"…

I think that's a drafting error though, and it did mean to say that IP
addresses MUST be represented as an iPAddress.

It does go on to make it clear that the dNSName isn't just any old
IA5String, but must be in the "preferred name syntax", with explicit
reference to RFC1034 §3.5. So no, you can't stick a textual
representation of an IP address in there, even if you're going to
completely ignore how broken that would be because even Legacy IP
addresses can be represented in non-canonical forms.

Either way, your find in RFC6125 makes it clear that implementations
MUST NOT accept certificates which have only a textual representation
of the IP address in their Subject CN or a dNSName SAN, when the IP
address has been used in the URI.

IP addresses are IP addresses. Domain names are domain names.

In X.509 you never compare one to the other.

If your caller gives you a "hostname" which is actually an IP literal,
it only makes sense to compare that with an iPAddress field in the SAN,
never anything else.