4

I am currently debugging a very old game (so there is no customer support) that keeps crashing under certain condition and I have found out it is because unsigned short to unsigned int conversion was done by signed conversion (possibly code similar to "short int x = 0x8000; unsigned int y = (unsigned int)x" which leads to y = 0xffff8000 instead of 0x00008000)

Now I have to debug the game in disassembly code to find out all the movsx command that does the wrong conversion and change them to movzx command (there are many places where signed conversion is correct too so I cannot just change all movsz to movzx).

The problem is, when I am at a crashing frame (where another function tries to access the address 0xffff8000, for example), that crashing function might be miles away from the original movsx code that does the conversion. The crashing function (and its callers) is called thousands of times so it is not possible to put a break point at some point before and see how the register value changes. It appears that, for every time I run the program, the stack address is different from the previous run, so data breakpoints does not work as well since I don't know which address to break on before the crashing happens.

I need a good way (a way that has 100% of chance to find the bug in relatively consistent time) to find out when does the value 0xffff8000 is pushed onto the stack. If someone with experience in asm debugging can give me some techniques to achieve this I will really appreciate.

EDIT: (Although the question in the title is still not answered, the following information I found, which helps me fixed the specific scenario, might be useful to someone in the future)

I have found out that Visual Studio gives the wrong call stack sometimes. It seems to find the callstack based on stack pointer value searching, so if a function start with something like "sub esp,2Ch" then the callstack will be screwed up.

To debug this issue one needs to manually set a breakpoint before the crash, with condition that the register is at the wrong value, and when it is hit one needs to manually modify the register to the correct value; after that execute the function until it ends and see where it returns to.

|improve this question
1

What about the call stack when the crash happens ?

In gdb you can get it with backtrace command.

You should be able to see the crashing function caller address and so on recursively.

You can check if before the fatal call happens, you can find the code performing the wrong conversion.

Edit

Is the memory location with the wrong data fixed ?

If so, watch it with the debugger and looking at the call stack try setting breakpoints to code locations getting executed before the fatal function.
You should pop in a point of the code flow where that memory location has not been set filled in with the wrong value yet.
Once you are there, while you are watching that code location, perform some Step Over and identify which call makes the memory location content change.
Once you have found it, put a breakpoint there, rerun everything and perform the same actions on that function inner code.
Repeating this iteratively you should converge to the location where the actual memory location modification happens.

Tools

IDA Pro (Disassembler) https://www.hex-rays.com/products/ida/support/download.shtml

OllyDbg (Debugger) http://ollydbg.de/

|improve this answer
  • Currently I am going down the callstack to try to find if there is any movsx prior to calls along the path, however in the problematic scenario there are recursive calls and the call stack is long; I have gone down at least ten levels and could not find the movsx nearby; some functions are extremely long and it is not possible to identify whether a movsx that is really far away from the call is the cause of the crash without setting a break point and see what really happens. – cr001 Aug 29 '16 at 13:43
  • Update the answer in the Edit Section – Nicola Bernini Aug 29 '16 at 13:52
  • The wrong data is passed through stack to the function; the assembly code looks like "mov esi,dword ptr [esp+40h]" where after this instruction esi becomes the bad value. However the immediate caller of the function does not have any movsx so it is likely the parameter is passed along the call-path. This is why I need to know where exactly in the call-path is the value pushed onto the stack. – cr001 Aug 29 '16 at 13:55
  • 1
    Maybe you are working with a too detailed view on the code: is your debugger able to perform some pattern matching so to recognize some C Standard Library Functions? I remember IDA Pro had some plugins providing this feature, which tools are you using? You should try to stay out of that highly repetitive library code and work at business logic level. – Nicola Bernini Aug 29 '16 at 14:14
  • 1
    I see, unfortunately I have not used it so I don’t know its features well. Maybe you should have a look at the asm with IDA Pro (imho on of the best disasm ever) as it should recognize C StdLib (check if it’s a built-in function or needs a free plugin). For Debugging you could try OllyDbg Look at the Edit for Links – Nicola Bernini Aug 29 '16 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.