I looked quickly through couple of links on the net:

http://archives.wnpvam.com/001/Broadcasting%20Software/071201/Manuals/AudioScience/SDK/asx_usermanual.pdf
http://www.audioscience.com/internet/download/sdk/asx_usermanual_html/html/group__System.html

In my opinion, the way of declaring and calling this function hardly can be different from what Sergey already suggested.

DECLARE INTEGER ASX_System_Create IN ASX32.DLL;
	INTEGER asxSystemType, INTEGER @phSystem
	
LOCAL phSystem, nResult
phSystem=0

nResult = ASX_System_Create(1, @phSystem)

I would look for some other reasons. For example, the manual says in "Installation and Usage" part

"1. Make sure you have an AudioScienceWAVE or WDMdriver installed
and working with at least one audio adapter..."

Missing driver can very much be a reason for this kind of error. Also may be try different values for the first parameter.

* * *
Still there is one more way you could try. Note that the second parameter in the declaration now is passed by value not by reference.

DECLARE INTEGER GlobalAlloc IN kernel32;
	INTEGER wFlags, INTEGER dwBytes

DECLARE INTEGER ASX_System_Create IN ASX32.DLL;
	INTEGER asxSystemType, INTEGER phSystem

LOCAL phSystem, nResult
phSystem = GlobalAlloc(0, 4)

nResult = ASX_System_Create(1, phSystem)

* * *
The original syntax is a usual way for nerdy C-people to make simple things look way too sophisticated:

typedef void* ASX_HANDLE -- means "redefine variable type void* as ASX_HANDLE"; in other words instead of typing "void*" one can type ASX_HANDLE

In C++ void* itself means a pointer that can point to any variable that is not declared with the const or volatile keyword. A void pointer can be converted into any other type of data pointer.