Hello,
The service could fail to start when using our "Services" page if the service application has some dependecies (e.g. a .NET assembly) that were not already installed before the "StartServices" action. The .NET assemblies are registered after the "StartServices" action during installation. In this case the correct approach will be to use a custom action like you already tried.
If the BAT file you already added in "Files and Folders" page contains the following command line:
then you should not duplicate this in the "Command Line" field of your "Launch File" custom action. Just leave the "Command Line" filed empty. Also, please uncheck the "Run as administrator" option of your "Launch file" action. Since the main setup has the "Run as administrator" option enabled in "Install Parameters" page it is redundant to enable the same option for your action too. When the "Run as administrator" option from "Install Parameters" page is used the whole installation process will run elevated from the beginning to end.
Finally, please go to "Services" page and delete the service control operation you configured to start your service on install.
Let us know if this helped.
All the best,
Daniel