Here is an email reply from tech support, which was very helpful.
Let me explain what you need to do:
-- first you will show the standard action Begin under InstallUISequence and InstallExecuteSequence in the "Custom Actions" page. Here is is explained how to do this:
http://www.advancedinstaller.com/user-g ... question38
-- now add the predefined custom action Detect Process and set the "Action Data" field to the exact name of your process, as you can see it with task manager
-- set the custom action condition to:
NOT Installed
-- now press [Shift] key and drag the custom action under the other Begin standard action; this will make the custom action to be shared between the two standard actions.
Now, to stop the installation if the process is running is very simple and you don't need to use another custom action. You can use our built-in support:
-- go to "Launch Conditions" page, "Custom Launch Conditions" tab
http://www.advancedinstaller.com/user-g ... -user.html
http://www.advancedinstaller.com/user-g ... ialog.html
-- here create a new launch condition and set the condition to:
AI_PROCESS_STATE = "Stopped"
-- the Description field can be set to a mesage that you wish to the display to your users; for example: "Please close application XR in order to correctly install this package."
This should be enough to accomplish the desired actions in case the process you are searching for it is running.