I built a rev challenge that turns into crypto after like 4 layers of pain. Am I evil?
I’ve been working on a CTF challenge called palimpsest, and I wanted to share the idea because it’s probably the most evil rev challenge I’ve made so far.
The public files are just a stripped ELF and an encrypted pack file. There’s no input prompt, no “correct/wrong” branch, and no plaintext flag. Running it normally just gives you a mostly useless digest, so the solve is not about patching a branch or finding a password check.
The binary has a couple fake VM paths and one real custom VM. The real VM uses stateful opcode decoding, so the same encoded instruction can mean different things depending on runtime state. There are also a few self-modifying bytecode spots, so static disassembly alone is kind of doomed. The intended move is to reverse the VM loop, write a tracer or emulator, and dump what it builds in memory.
That memory dump is not the flag either. It’s a masked typed IR that turns into a modular constraint graph. After decoding the IR and simplifying the constraints, the actual crypto finally shows up: a custom Schnorr-style signature system with decoy samples and a noisy affine nonce relation.
So the solve path is basically:
- unpack the sections
- ignore the fake VMs
- reverse the real VM
- write an emulator/tracer
- dump the hidden IR
- decode/type-infer the IR
- simplify the constraint graph
- extract the Schnorr equations
- filter decoys
- run a lattice attack
- recover the private scalar
- decrypt the flag
I’m trying to keep it brutal but fair. No random crashes, no anti-debugging junk, no network stuff, and no impossible crypto. The hardness is supposed to come from the layers and the fact that each stage only reveals the next representation.
Curious what rev people think: is this the fun kind of evil, or did I cross into “I am switching to OSINT and finding where the challenge developer lives” area?