I have been able to find the trigger for this.

When this occurs, the latency is caused by the Service Control Manager, more specifically with the Smart Card Device Enumeration service.

In the attached image, this generates three errors. The first one is the RDP access itself, timing out after 30 seconds. Then, it is followed by two Smart Card Device Enumeration service errors.

The sequence is as follow:

1. A timeout (30000 milliseconds) was reached while waiting for a transaction response from the UmRdpService service.
2. A timeout (30000 milliseconds) was reached while waiting for a transaction response from the ScDeviceEnum service.
3. The Smart Card Device Enumeration Service service failed to start due to the following error: The service did not respond to the start or control request in a timely fashion.