Correct - because IIS accesses the file through the Network User you have to allow the Network User access to the directory or the file you want to access through IIS.
The Network User is not part of the list when you view the users under computer management console. That is why you need to access the file or directory through Windows Explorer, right-click on the the folder or file, select sharing & securiy, select the security tab, click on the add button, type in Network, click on Check Names Button and choose Network (not Network Service).
Once you add the user you can give whatever rights are necessary.
Now, if the files reside on another machine you have a whole different scenario you need to go through...