This is resolved by adding a registry key on the remote server.

This is under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout and we need to add a DWord key under IgnoreRemoteKeyboardLayout with a value of 1.

Then, if a keyboard is in effect, simply remove it, terminate the connection and get back at it. The keyboard layout from your PC should no longer be broadcasted to the Terminal Server.