Re: Crc32

[Andrew Fish <afish@apple.com>]

> On Aug 30, 2016, at 8:17 AM, valerij zaporogeci <vlrzprgts@gmail.com> wrote:
> 
> well, that bit in the 32th (bit 33) term of the polynomial is
> "implied", but it should be used, otherwise it would not give a 32 bit
> remainder (crc). if to count 04c11db7 as a polynomial, its non-zero
> msbit is bit 26, so with it, it would be crc25.
> Why results don't match, I suspect it's becuase that pre- and post-
> processing involved. Tianocore algorithm inverses bits at the
> beginning and in the end.
> Pure definition of CRC doesn't require this. It would be good to know
> the exact specification on the algorithm and its nuances.
> And of course in the pure case, calculating CRC on the polynomial
> itself should give 0 (since polynomial itself is multiple of itself).
> It's the easiest way to check. So CRC of 104c11db7 = 0
> In the the above calculator (in the Michael's response), neither
> CRC(104c11db7) nor CRC(04c11db7) gives 0. (4E24FFBE and B79B7E7A
> respectively).
> So if it is a right CRC32, then it's not a pure CRC mathematically.
> and additionally processed. How? and where it is possible to learn
> that?
> 

I don't know the math here as well as you do so I will not comment on that.  My guess is this is an old algorithm that was optimized for 32-bit hardware. 64-bit math would require a lot software math libraries and it would be much slower than doing 32-bit math on a processors ALU. 

If you want to research the history of this algorithm your best bet it to research the CRC32 used in old school Ethernet. The CRC got added to the (U)EFI spec about 17 years ago and it was probably copying something that was 25-30 years old, but that is just my best guess as Ken called in rich a long time ago and does not work on things EFI at this point. 

Thanks,

Andrew Fish

> 
> 2016-08-30 17:09 GMT+03:00, Andrew Fish <afish@apple.com>:
>> 
>>> On Aug 30, 2016, at 6:32 AM, valerij zaporogeci <vlrzprgts@gmail.com>
>>> wrote:
>>> 
>>> Thank you.
>>> But the polynomial value for CCITT crc32 is 104c11db7, not its tail
>>> 04c11db7.
>> 
>> The spec does not say it is a CCITT, it states it uses the CCITT algorithm
>> with a seed polynomial of 0x04c11db7.
>> 
>> 
>>> And this whole mess with all those online calculators was the exact
>>> reson I asked here. Because they mostly give WRONG result. As your
>>> example. It's easy to check it's wrong. Because for example leading
>>> zeroes in the message MUST not change the Crc32.
>>> Here we have:
>>> for 01 -> CD39D477
>>> for 00 01 -> D71E16AC
>>> and so on.
>>> Also,  it's obvious that Crc32(1) would be a polynomial without the
>>> most significant non-zero bit, thus crc32 from 1 by the polynomial
>>> 104c11db7 would be 04c11b7.
>>> The only online calculator so far giving right results is here:
>>> https://ghsi.de/CRC/index.php?Polynom=100000100110000010001110110110111&Message=0104c11db7
>>> 
>>> I wrote function following the mathematics of the Crc and it gives
>>> results exactly like in the above calculator.
>>> It's fun to imagine what happens with such an incompatibilty for
>>> example in case of GPT header.
>>> 
>> 
>> As far as I can tell this is used in a lot of real world things: HDLC, ANSI
>> X3.66, ITU-T V.42, Ethernet, Serial ATA, MPEG-2, PKZIP, Gzip, Bzip2,
>> PNG,[36] many others[1]
>> 
>> I think Ken added this to the spec in the last millennium. Ken's background
>> was networking and Windows X86 HAL so I'm assuming he picked the same CRC as
>> Ethernet?
>> 
>> It is always possible we could have botched the math some place along the
>> way and not noticed as we are self consistent so everything appears to
>> work.
>> 
>> Thanks,
>> 
>> Andrew Fish
>> 
>> [1] https://en.wikipedia.org/wiki/Cyclic_redundancy_check
>> 
>> 
>>> 2016-08-30 14:08 GMT+03:00, Michael Zimmermann
>>> <sigmaepsilon92@gmail.com>:
>>>> well as u already said, the spec says it uses 'a standard  CCITT32 CRC
>>>> algorithm with a seed polynomial value of 0x04c11db7'
>>>> 
>>>> this is the implementation which confirms it:
>>>> https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/RuntimeDxe/Crc32.c
>>>> 
>>>> after testing it it indeed produces CCITT32 results like this online
>>>> generator:
>>>> http://g6auc.me.uk/CRC32/index.html
>>>> 
>>>> Thanks
>>>> Michael
>>>> 
>>>> On Tue, Aug 30, 2016 at 2:54 AM, valerij zaporogeci
>>>> <vlrzprgts@gmail.com>
>>>> wrote:
>>>> 
>>>>> Hi, all.
>>>>> Yet another dumb question from me.
>>>>> UEFI specification has Crc32 calculation service and uses Crc32 in
>>>>> several places. but it only humbly mentions in the note somewhere in
>>>>> the description of System Table about what exact one it wants. Namely
>>>>> it states that the polynomial seed is 04c11db7. And that's all.
>>>>> My question is - does really the specification means the 33-bit
>>>>> polynomial
>>>>> 104c11db7? And is the algorithm just a plain remainder calculation
>>>>> without any additional pre/post processing of it? So that for example
>>>>> Crc32 of the 4-byte sequence b16b00b5 would be
>>>>> 8c1f0a7c?
>>>>> Thank you.
>>>>> _______________________________________________
>>>>> edk2-devel mailing list
>>>>> edk2-devel@lists.01.org
>>>>> https://lists.01.org/mailman/listinfo/edk2-devel
>>>>> 
>>>> 
>>> _______________________________________________
>>> edk2-devel mailing list
>>> edk2-devel@lists.01.org
>>> https://lists.01.org/mailman/listinfo/edk2-devel
>> 
>> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel