Custom Action failing to run in environment with McAfee HIPS protection

[Koomazaz]

Hello,

I have three Advanced Installer projects that have successfully built and deployed several versions of our software. All three installers use Custom Actions and access C# methods stored in external dll files. All three installers are built properly and run successfully on our testing and development environments. The custom actions have been called and executed successfully.

We have recent built new testing environments containing McAfee Host Intrusion Prevention for Server (HIPS). We developers have no control over configuring HIPS, and several of our clients have started using it since we deployed our last version of our software.

On the newly-created VM with HIPS, we are seeing that one of our three installers is now failing when it tries to run a custom action. The other two installers contain similar custom actions which confirm database connectivity, and the other two installers can successfully install. To clarify the situation, each installer has its own CustomAction dll containing specific logic for the installer, and each of those may reference InstallerCommon dll which contains shared methods. All three installers work on our old VMs without HIPS installed. Two of our installers work on the new VMs with HIPS.

The installer that is failing is our Database Server installer. When it calls a "TestDbConnection" custom action, we get the following lines in the log:

Code: Select all

Action start 16:23:49: TestDbConnection
MSI (c) (D4:08) [16:23:49:835]: Invoking remote custom action. DLL: C:\Users\(username)\AppData\Local\Temp\MSI570C.tmp, Entrypoint: TestDbConnection MSI (c) (D4:08) [16:23:49:835]: Cloaking enabled.
MSI (c) (D4:08) [16:23:49:835]: Attempting to enable all disabled privileges before calling Install on Server MSI (c) (D4:08) [16:23:49:835]: Connected to service for CA interface Action ended 16:23:51: TestDbConnection. Return value 3.

For one of the other two installers, the Application Server installer, we call a custom action that does the exact same DB check, and we get the following:

Code: Select all

Action start 16:23:49: TestDbConnection
MSI (c) (D4:08) [15:28:14:322]: Invoking remote custom action. DLL: C:\Users\(username)\AppData\Local\Temp\MSI71CB.tmp, Entrypoint: TestDbConnection MSI (c) (D4:08) [15:28:14:322]: PROPERTY CHANGE: Modifying OverrideValidDatabaseServerCheck property.  Its current value is 'false'. Its new value: 'true'
Action ended 15:42:51: TestDbConnection. Return value 1.

In addition, I have put debugging messages (pop-up Message boxes) inside the Custom Action, and they do not pop up on the Database Server installer, implying that we are not permitted to enter the custom action at all.

The problem is that these custom actions all run on an environment without HIPS. On an environment with HIPS, only two of the three succeed.

The HIPS client log provides a message:

Code: Select all

Attack type: DISA McAfee - Prevent DLL files from Running in User Profile (Sig Id = 7020)

The HIPS administrator provided the additional information:

Code: Select all

Event Description
C:\WINDOWS\SYSTEM32\RUNDLL32.EXE running with the privileges of user (COMPUTERNAME)\admin on the system with Agent (COMPUTERNAME) attempted to open file/directory C:\Users\(username)\AppData\Local\Temp\2\MSI4CD6.tmp-\Microsoft.Deployment.WindowsInstaller.dll in order to:
read
execute

We are absolutely puzzled. It seems like it could be an environment issue, except for the fact that two of our installers are able to extract the dlls and run them from the AppData\Local\Temp directory. That implies that it might be an installer problem, except for the fact that all three work on our older VMs, and the custom actions and AIPs are extremely similar.

Any insight or advice would be welcome! Thanks.

[Daniel]

Hello and welcome to our forums,

First of all I apologize for my delayed reply. I'm afraid I'm not aware of any settings you can use to avoid this detection. Most likely this is a McAfee false positive detection of your DLL and I think you should try contacting them to report this so they fix it.

All our custom action are using the standard Windows Installer action types. At a first Glance it seems McAfee falsely detect the temporary copy of your DLL which is created at install time when the custom actions runs. This is the Windows Installer custom action behavior when using the "Call function from attached native DLL" action type. Before the DLL function is called Windows Installer creates a temporary copy in %temp% of the target DLL.

All the best,
Daniel