Plateforme Level Extreme
Abonnement
Profil corporatif
Produits & Services
Support
Légal
English
What's wrong with CRC32
Message
De
06/01/2014 19:11:15
 
 
À
Tous
Information générale
Forum:
ASP.NET
Catégorie:
Autre
Titre:
What's wrong with CRC32
Versions des environnements
Environment:
VB 9.0
OS:
Windows 7
Network:
Windows 2003 Server
Database:
MS SQL Server
Application:
Web
Divers
Thread ID:
01591437
Message ID:
01591437
Vues:
83
This is something interesting. I am not sure if someone knows something about it.

Recently, we moved into a new environment. Thus, the share drives are no longer Windows recognized drives but a NAS.

There is one NAS device for the virtual environment, where, acting as a VMWare, allows us to have various share drives on one which might differ from the other one, as well as some specific configurations to each of them.

The NetApp filers contain two environments, known as virtual filers. Both share the same physical interfaces, same memory, same heads and same cache.

The NAS hardware being used is from NetApp company. They use a V-Series controller. They use disk rack called Disk Shelves. The data protocol is FC at the physical layer. This is like a special TCP. On top of FC, the protocol used is CIFS. This is what Windows uses.

Since we are in that environment, we just discovered that the CRC32 returned for a JPG file no longer matches the CRC32 returned from the old environment. So, we are wondering if the .NET CRC32 being used is obtaining a different value from one technology to another.

It is noted that at the same time, we also moved to .NET Framework 4.5. So, we are not sure if the new .NET Framework DLL are responding differently and that could be the source of the problem.

Presently, this is not good. The production environment ends up with different values for the CRC32 where duplicate files should be recognized and they are not.

Anyone has seen something like this before? Either the NAS technology is responsible for returning different CRC32 values or that we have new .NET DLL and this is something that I wouldn't be able to explain either.

In case someone would worry, here is the CRC32 class of my framework:
Imports System.IO

Public Class CRC32

    Public cMessage As String = ""
    Public cFileName As String = ""
    Public lLogError As Boolean = True
    Public nCRC32 As Integer = 0
    Public oApp As Framework.App = Nothing
    Private cFileDoesNotExist As String = ""
    Private nBufferSize As Integer = 1024
    Private nCRC32Table() As Integer
    Private nLanguage As Integer = 0
    Private oProcess As Framework.LXProcess = Nothing

    ' This is when we access the class in a desktop mode
    Sub New(ByVal toApplication As Framework.App)
        oApp = toApplication
        nLanguage = oApp.nLanguage
        Init()
        Initialize()
    End Sub

    ' This is when we access the data provider in a Web mode
    Public Sub New(ByVal toProcess As Framework.LXProcess)
        oProcess = toProcess
        oApp = oProcess.oApp
        nLanguage = oProcess.nLanguage
        Init()
        Initialize()
    End Sub

    Private Function Init() As Boolean

        ' Based on the language
        Select Case nLanguage

            ' English
            Case 1
                cFileDoesNotExist = "The file ##File## does not exist."

                ' French
            Case 2
                cFileDoesNotExist = "Le fichier ##File## n'existe pas."

                ' Spanish
            Case 3
                cFileDoesNotExist = "The file ##File## does not exist."

                ' Portuguese
            Case 4
                cFileDoesNotExist = "The file ##File## does not exist."

        End Select

        Return True
    End Function

    ' This is the official polynomial used by CRC32 in PKZip.
    ' Often the polynomial is shown reversed (04C11DB7).
    ' Initialize
    Private Function Initialize() As Boolean
        Dim lnDWPolynomial As Integer = &HEDB88320
        Dim lnCounter As Integer = 0
        Dim lnCounter2 As Integer = 0
        ReDim nCRC32Table(256)
        Dim lnDWCRC As Integer

        For lnCounter = 0 To 255
            lnDWCRC = lnCounter

            For lnCounter2 = 8 To 1 Step -1

                If (lnDWCRC And 1) Then
                    lnDWCRC = ((lnDWCRC And &HFFFFFFFE) \ 2&) And &H7FFFFFFF
                    lnDWCRC = lnDWCRC Xor lnDWPolynomial
                Else
                    lnDWCRC = ((lnDWCRC And &HFFFFFFFE) \ 2&) And &H7FFFFFFF
                End If

            Next

            nCRC32Table(lnCounter) = lnDWCRC
        Next

        Return True
    End Function

    ' This is the main method to call
    Public Function GetCRC() As Boolean
        Dim lcMessage As String = ""
        Dim lnCounter As Integer = 0
        Dim lnDelay As Integer = 250
        Dim lnRetry As Integer = 8
        Dim loStream As Stream = Nothing

        ' Reset the values
        cMessage = ""
        nCRC32 = 0

        ' We retry up to 8 times
        For lnCounter = 1 To lnRetry

            Try

                ' If the file exists
                If oApp.FileExist(cFileName) Then
                    loStream = New FileStream(cFileName, FileMode.Open, FileAccess.Read, FileShare.Read, 8192)
                Else
                    lcMessage = oApp.StrTran(cFileDoesNotExist, "##File##", cFileName)

                    ' If we log the error
                    If lLogError Then

                        ' Get the proper definition as per the current scope
                        If oProcess Is Nothing Then
                            oApp.ErrorSetup(, lcMessage)
                        Else
                            oProcess.ErrorSetup(, lcMessage)
                        End If

                    Else
                        cMessage = lcMessage
                        Return False
                    End If

                End If

                Exit For

            Catch loError As Exception

                ' If we have not reached the maximum tries of 8, we retry
                If lnCounter < 8 Then

                    ' Wait for a timeout before retrying
                    System.Threading.Thread.Sleep(lnDelay)

                    Continue For
                End If

                ' If we log the error
                If lLogError Then

                    ' Get the proper definition as per the current scope
                    If oProcess Is Nothing Then
                        oApp.ErrorSetup(loError)
                    Else
                        oProcess.ErrorSetup(loError)
                    End If

                Else
                    cMessage = loError.Message
                    Return False
                End If

            End Try

        Next

        nCRC32 = GetCrc32(loStream)
        loStream.Close()
        Return True
    End Function

    Private Function GetCrc32(ByRef toStream As System.IO.Stream) As Integer
        Dim lnCRC32Result As Integer = &HFFFFFFFF
        Dim loBuffer(nBufferSize) As Byte
        Dim lnReadSize As Integer = nBufferSize
        Dim lnCount As Integer = 0
        Dim lnCounter As Integer = 0
        Dim lnLookup As Integer = 0

        lnCount = toStream.Read(loBuffer, 0, lnReadSize)

        Do While lnCount > 0

            For lnCounter = 0 To lnCount - 1
                lnLookup = (lnCRC32Result And &HFF) Xor loBuffer(lnCounter)

                ' Nasty shr 8 with vb :/
                lnCRC32Result = ((lnCRC32Result And &HFFFFFF00) \ &H100) And 16777215

                lnCRC32Result = lnCRC32Result Xor nCRC32Table(lnLookup)
            Next

            lnCount = toStream.Read(loBuffer, 0, lnReadSize)
        Loop

        Return Not lnCRC32Result
    End Function

End Class
That class remained unchanged since a very long time. So, this is the same code as before.
Michel Fournier
Level Extreme Inc.
Designer, architect, owner of the Level Extreme Platform
Subscribe to the site at https://www.levelextreme.com/Home/DataEntry?Activator=55&NoStore=303
Subscription benefits https://www.levelextreme.com/Home/ViewPage?Activator=7&ID=52
Suivant
Répondre
Fil
Voir

Click here to load this message in the networking platform