>SQL Server constantly monitors page usage in Data cache buffers and freed pages that are not in use recently when free pages pool size drops below minimum. Any read operation that uses Data cache buffers could cause already cached pages to be freed.

Ok, so in our case, the absence of hits during the overnight was probably in cause of the caching being dropped from the related table. Then, it became fast again on the upcoming hits. Also mentioned, the backup that happened two hours before might have been a factor. I assume that it required significant usage of SQL Server engine and might have caused the same situation as well.