I am not sure it this could help but it cannot hurt, I am using the Using\End Using approach to encapsulate the StreamReader class:

                        ' FTP setup
                        loFtpWebRequest = System.Net.FtpWebRequest.Create(New Uri(lcURI))
                        loFtpWebRequest.Credentials = New System.Net.NetworkCredential(cUsername, cPassword)
                        loFtpWebRequest.Method = System.Net.WebRequestMethods.Ftp.ListDirectoryDetails

                        ' If we use FTP/SSL  (AUTH SSL)
                        If nNoFTPProtocol = 2 Then
                            loFtpWebRequest.EnableSsl = True
                        End If

                        loFtpWebResponse = loFtpWebRequest.GetResponse()

                        Using loStreamReader As New StreamReader(loFtpWebResponse.GetResponseStream())

                            ' Load the files into the string builder
                            loStringBuilder.LoadString(loStreamReader.ReadToEnd())

                        End Using

This is the part that deals with memory so maybe this could have been a factor.