Ransome,

There should be nothing to reconfigure on the original two workstations. However make sure that the router you choose is configureable as a DHCP server.
Make sure you choose one that is compatible with your broadband connection. Some of these come with four or eight ports which are enough for your network.

If all four machines access the router at the same time, your network speed will be one-fourth the speed of one machine accessing it.

When you do this make sure that the workstations are set to receive their TCP/IP addresses automatically.

Mike