>how much memory does the OS need? I guess I need to monitor that and not let the memory spill into the swap file, right?

There's actually a formula but I don't have it memorized.

>If I let SQL Server manage it dynamically, will it start using the swapfile? If so, is that a preferred practice over limiting the memory to installed RAM?

You do not want SQL Server hitting the swap file. Nor do you want the OS hitting it. SQL Server's dynamic allocation will handle this.

-Mike