Dammit! Never tell him you're going to take over until you've stormed the stronghold, and he's still completely unaware of the situation. Then you gloat, with him completely unaware of your control over the mighty castle. Just as he calls his guards, have your men come in and announce that his guards are all either dead, or converted!
---
Do you want ants? Because that's how you get ants.