Broaching is a technique for turning a round hole into a square hole or a hex hole or for adding a keyway into a hole. A broach is a toothed tool that looks a bit like a rack or coarse tooled file. However the teeth are different sizes as you go along the broach. The idea is that each tooth of the broach sticks out just a little bit more than the next and hence can shave a small amount off the corners until the last tooth is reached which is the desired size of the hole. Broaches are quite expensive to purchase and are easy to break. Here's some tips on how to make them and a device that pulls rather than pushes the broach meaning that it's less likely to bend.
31/10/2010 links updated
Graham Howe's Workshop Projects
Broaching Practice Book
Home made keyway broach made by grinding down old taps.
Building GWR Beetle
2 hours ago