>How would you detect that a user clicked in the cancel button in the ColorDialog dialog?

The ShowDialog() method returns a DialogResult enum value which you can check:

if ( myColorDialog.ShowDialog() == DialogResult.Cancel)
{
//User Cancelled......
}