I had to drill some steel plate yesterday and even though I was using a vice and pillar drill (drillpress) the drill started to wander as it was a little bendy.
This happened twice, once on a through hole and once on a blind hole.
My first tip is for that through hole. Simply stop, take the work out the vice, turn it over, remark the hole, centre punch and start again. This worked just fine for me.
The second item was a little more challenging. I needed to have a blind hole which was going to be tapped and the drill moved again. This time I could not turn over the work so I switched to a centre drill, put more clamps on the vice and gently drilled a new hole with the centre drill. This was followed up with the origional 4.5mm drill.
There will be some pictures of this when I do the write up on the knurling tool.