There's a book out there that talks about MTS 2.
Publisher: Sam's
Author: Roger Jenning
Title: (Something) Microsot Transaction Server 2.0
Sorry, can't remember the exact title.
I played with it a little bit (not in production) with VB Dlls and it was working fine.
Are you using the Context object ?
Are you breaking your transaction in it's smalest elements ?
Are each part of the transaction stateless (each part not aware of the others) ?