My Investigation of MOPS Crashes

After several years of running smoothly, our installed MOPS application shows an intermittent crashes with error message as follows:

mc01

There are no crash dump whatsoever, and the event of this crash occurred unpredictably. The program that crashes, as shown in the error message box is mhistsp.exe.

This program is part of MOPS infrastructure to supply tag data information from MOPS Historical Data to the WinMOPS client. The process of supplying information is done at regular interval, such as every one minute.

Since there are no crash files from the OS, I decided to use the DebugDiag from Microsoft. After perform settings for appropriate configuration and waits for another instance of crash event, I’ve found that there are actually more than one crash dump files for each crash event.

Without any source code, it is certainly a daunting task, even with only one crash files. So, after spending hours analyzing each files, it is revealed that only one crash dump file that matters, the other is caused or cascaded from the first event of crash. By examining the time stamp of each crash dump files, I can then determine the correct one to be analyzed.

Here’s the call stack of the crash event in question:

mc02

From here, it is obvious that mhistsp.exe throws a C++ exception. By examining the data structure and routines just before the crash event occurred, it is revealed that mhistsp.exe checks for the VARIANT variable, when it contains an array. The problem is, for this particular version of MOPS’s mhistsp.exe, it only accepts array of VARIANT variable, where as at the crash event, the VARIANT variable contains array of something else.

Based on these information, I can then perform the configuration of DebugDiag script to gather register variables and the callstack at the point before mhistsp.exe is throwing C++ exception, and also instruct DebugDiag to create the crash dump files. I will then have 2 files at my disposal, one is the register variable for data structure analysis and the other is actual crash dump files.

The array of something else, reveals to be array of VT_UI1. But the question is, what tag id that has this kind of weird data structures that will be rejected by MOPS’s mhistsp.exe ? This, in itself is also a daunting task, which requires performing more detailed analysis of the parameter passed to ReadTagData method.

In the end, I’ve managed to retrieve tag id that causes the crashes, and after applying the changes to MOPS’s tag data structures, the crash is not happening anymore 🙂

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: