In order to achieve this, you can use the first 4 steps from the Custom Launch Condition Help thread, but use a "Display error message" custom action instead of the "Display message box" custom action.
I use this method for my application (module name Spartan.exe). When Windows 10 is released, it will ship with a new browser which will also have the module name Spartan.exe. I would be most grateful if anyone could point me in the right direction to distinguish whether the detected module is my Spartan.exe or Microsoft's Spartan.exe
I'm afraid that our predefined custom action cannot make any difference between two processes with the same name. In order to achieve this, you can create your own custom action which will detect and close your application based on some specific attributes of it.