Starting out as I always do, I ran
strings, etc, then ran in my VM in
gdb and opened the binary in IDA Pro. The first thing I noticed is that IDA can’t view main in graph mode – They are likely jumping around weird to trick disassemblers.
Looking in IDA, a few instruction into main there’s a jump back a few bytes, into the middle of another instruction – suspicions confirmed. That’s no problem. Hit
D on the instruction that overlaps with the jump target to turn the code into data, move to the correct byte, and then hit
C to start disassembling from there.
That makes more sense, but there’s another jump into the middle of some instructions. You’ll find this a lot in this binary, but you can just keep hitting
C to fix it up. Since the jumps go backwards, you’ll be disassembling ‘over’ some of the jumps, but because they’re only there to confuse you, you can pretty much just follow the code straight down.
After doing that cleanup twice we see a reference to the first string we see “Please Enter the valid key!\n”.
Continuing to clean as you proceed down, there’s a call to
strlen ensuring the length is 16 bytes. I like to put breakpoints on checks like these and restart the program to validate, but otherwise do most of my work statically.
The rest of the check is straight forward: It checks that the xth byte is some hard code value, and then checks that the value of the xth byte from the back, plus the xth byte adds up to some other hardcoded value. You can hit
R on the byte values to see the ASCII representation, but this won’t work on the back half checks, because the two characters’ sums aren’t likely still in the ASCII range. After 8 of these double checks you should have what you need.