>I have a FTP class that is being used widely and it has a weird issue when I download a big file. Basically, this has been running for weeks. But, this big backup file I want to automate and have on my PC is not fitting very well within the actual code. Here is the FTP.Download() method I am using. I know it could be a little bit difficult to someone to try to see what could be enhanced to support that but here is is:
>
>
> ' Download a file
> Public Function Download() As Boolean
> Dim lcMinute As String = ""
> Dim lcURI As String = ""
> Dim llSuccess As Boolean = False
> Dim lnActual As Long = 0
> Dim lnActualABS As Long = 0
> Dim lnBytesRead As Integer = 0
> Dim lnContentLength As Long = 0
> Dim lnElapse As Integer = 0
> Dim lnEnd As Double = 0
> Dim lnLast As Integer = 0
> Dim lnPercentage As Integer = 0
> Dim lnDownloaded As Long = 0
> Dim lnRemaining As Long = 0
> Dim lnStart As Double = Date.Now.Ticks
> Dim loBuffer(nBuffer) As Byte
> Dim loDownloadedData(0) As Byte
> Dim loFileStream As FileStream = Nothing
> Dim loFtpWebRequest As System.Net.FtpWebRequest = Nothing
> Dim loFtpWebResponse As System.Net.FtpWebResponse = Nothing
> Dim loProgressBar As Framework.ProgressBar = New Framework.ProgressBar(oApp)
> Dim loMemoryStream As MemoryStream = New MemoryStream()
> Dim loStream As Stream = Nothing
> Dim loStreamReader As StreamReader = Nothing
>
> ' Reset the values
> cError = ""
> lError = False
>
> Try
>
> ' If we have an originator
> If Not oOriginator Is Nothing Then
> loProgressBar.oOriginator = oOriginator
> End If
>
> ' Initialize the progress bar
> loProgressBar.ShowProgress(0, cDownloading + " " + cLocalFile + "...")
>
> ' URI
> lcURI = "ftp://" + cHost + "/" + cRemoteDirectory + "/" + cRemoteFile
>
> ' Because there is a bug in the .NET Framework with the use of
> ' System.Net.WebRequestMethods.Ftp.DownloadFile which will always return -1,
> ' we have to use a two step process to achieve this method
>
> ' FTP setup
> loFtpWebRequest = System.Net.FtpWebRequest.Create(New Uri(lcURI))
> loFtpWebRequest.Credentials = New System.Net.NetworkCredential(cUsername, cPassword)
> loFtpWebRequest.Method = System.Net.WebRequestMethods.Ftp.GetFileSize
> loFtpWebRequest.UseBinary = True
>
> loFtpWebResponse = loFtpWebRequest.GetResponse()
>
> ' Get the length
> lnContentLength = loFtpWebResponse.ContentLength
>
> loFtpWebResponse.Close()
>
> ' FTP setup
> loFtpWebRequest = System.Net.FtpWebRequest.Create(New Uri(lcURI))
> loFtpWebRequest.Credentials = New System.Net.NetworkCredential(cUsername, cPassword)
> loFtpWebRequest.Method = System.Net.WebRequestMethods.Ftp.DownloadFile
> loFtpWebRequest.UseBinary = True
>
> loFtpWebResponse = loFtpWebRequest.GetResponse()
>
> loStream = loFtpWebResponse.GetResponseStream()
>
> ' While there is something to download
> While True
>
> ' Prevent application from crashing
> System.Windows.Forms.Application.DoEvents()
>
> ' Try to read the data
> lnBytesRead = loStream.Read(loBuffer, 0, loBuffer.Length)
>
> If lnBytesRead = 0 Then
>
> ' Nothing was read, finished downloading
> System.Windows.Forms.Application.DoEvents()
>
> Exit While
> Else
>
> ' Write the downloaded data
> loMemoryStream.Write(loBuffer, 0, lnBytesRead)
>
> ' Increment the number of uploaded bytes
> lnDownloaded = lnDownloaded + lnBytesRead
>
> lnActual = lnDownloaded / lnContentLength * 100
>
> lnActualABS = Math.Abs(lnActual)
>
> ' If the last is not the same as the actual
> If lnLast <> lnActual Then
>
> lnElapse = ((Date.Now.Ticks - lnStart) / 10000000)
>
> lcMinute = oApp.SecondToFormat(lnElapse)
>
> lnRemaining = lnContentLength - lnDownloaded
>
> ' Update the progress bar
> loProgressBar.ShowProgress(lnActual, _
> cDownloading + " " + cLocalFile + "..." + oApp.cCR + _
> oApp.GetFormatValue(lnDownloaded, ",") + " bytes " + cOf + " " + _
> oApp.GetFormatValue(lnContentLength, ",").ToString + " - " + cRemaining + " " + _
> oApp.GetFormatValue(lnRemaining, ",").ToString + " bytes - " + lcMinute)
>
> Else
>
> ' Update the progress bar
> loProgressBar.ShowProgress(lnActual)
>
> End If
>
> lnLast = lnActualABS
>
> End If
>
> End While
>
> ' Elapse time it took for the download
> nSecond = lnElapse
>
> ' Convert the downloaded stream to a byte array
> loDownloadedData = loMemoryStream.ToArray()
>
> loMemoryStream.Close()
> loStream.Close()
> loFtpWebResponse.Close()
>
> ' Create the file
> Try
> loFileStream = New FileStream(cLocalFile, FileMode.Create)
> loFileStream.Write(loDownloadedData, 0, loDownloadedData.Length)
> loFileStream.Close()
> Catch loError As Exception
> lError = True
> cError = cUnableToCreateLocalFile
> Exit Try
> End Try
>
> llSuccess = True
> Catch loError As Exception
> lError = True
> cError = loError.Message
> End Try
>
> ' Make sure to remove it
> loProgressBar.Dispose()
>
> ' Force a refresh
> Windows.Forms.Application.DoEvents()
>
> Return llSuccess
> End Function
>
>
>Basically, the file I am downloading is 2.2 GB. That usually takes 2.5 hours with WSFTP. Now, I want to handle that from an application. So, I am using my FTP class for that. After about 20 minutes, I have this memory exception thrown error that stops the process.
>
>I think the loMemoryStream, declared with the MemoryStream(), cannot handle too much in memory. Basically, after about 20 minutes, I would say the download is about between 200 MB to 400 MB transferred. So, I am assuming that I would have to change my approach to negotiate with such a big files. Any idea?
I see two problems .
(1) As you say, you keep all of the 2.2 GB in a memory stream
(2) On top of that, you create another array of 2.2 GB and passing that to the FileStream
The solution that does not take extra memory is to write loBuffer to a FileStream directly ( as opposed to loMemoryStream)
Update: I do not see what the size of loBuffer is - but 4Kb to 16Kb is sufficient
Update 2: using passive mode is not a bad idea
> loFtpWebRequest = System.Net.FtpWebRequest.Create(New Uri(lcURI))
> loFtpWebRequest.Credentials = New System.Net.NetworkCredential(cUsername, cPassword)
> loFtpWebRequest.Method = System.Net.WebRequestMethods.Ftp.DownloadFile
> loFtpWebRequest.UseBinary = True
loFtpWebRequest.UsePassive= True
Gregory
