My son will be 5 on Wed and uses a pacifier to fall asleep at night sometimes..?

He uses it about 3 nights a week and falls asleep in 5 minutes. About 5 minutes after that I go in and take it and he doesnt notice. I am throwing it out the day after his birthday because we agreed he is too old for it. I feel bad taking it from him because I know it soothes him and it really isnt that big a deal to me if he has it 10 minutes out of the day. Am I wrong to take it from him?