Green Jumper Trouble?

I've been re-training my ex-racer as a hunter/jumper. He finally settled down enough and we've started jumping, but anyway he does wonderful on the ground with me or lunging. So I decided to trot him over some little ones under saddle but as soon as i give him rein to jump he swerves to avoid them. It doesn't matter what i do. Even if i keep him square to the jumps he just runs into them. It's ridiculous. And yes before you ask I have jumped many horses before, but never had one do this. So my question to you is- Why does he do this, but only under saddle? and How can I correct it? Thanks