Why didn't Peter Pan immediately kill all the Charmings when they stepped foot in Neverland?

It's been a while since I watched, so maybe they explained it, but if he's so powerful why couldn't he just kill them while they slept or something?

He wanted Henry to think they abandoned him, and what better way than to kill them, and make sure they never come for him?