Brake Booster Bracket Question

[xludan]

I refinished my brake booster this weekend and I have a question about putting it back together. I'm sorry if this is a little confusing. Hopefully the picture below helps to explain my problem. When I put the bracket that the booster rod and the brake pedal rod connect to back together it looks like the two middle pieces arent centerd anymore. There seems to be a little raised collet on the shaft that the brackets are supposed to rotate on that is keeping the rod brackets from being centered. Am I supposed to pound the rod bracket onto this collet, I'm assuming this wont let it rotate freely anymore? Any help?