Apparently, this seems to be a normal behavior.
https://docs.microsoft.com/en-us/sql/t-sql/functions/openrowset-transact-sql?view=sql-server-ver15When accessing remote OLE DB data sources, the login identity of trusted connections is not automatically delegated from the server on which the client is connected to the server that is being queried. Authentication delegation must be configured.
It is not clear however how to do this securely. I need to avoid dumping the Excel files on the database server directly. This has to remain on a network drive.