When to give up the pacifier?? Help!?

So my son is 2 years and 2 months now, he still uses his pacifier during the day, not every second of the day but a good amount. I'm not sure when to make him give it up. I hear SOOO many different opinions, some people will frown upon it and some people say who cares he'll give it up when he's ready. What is your experience with this situation? A little background though for you to think about, my son isn't as up to par in the talking, understanding everything just yet, he says a handful of words, doesn't use the potty, he's not "slow" just hasn't caught up just yet so taking it away now would traumatize him to no end. What are your thoughts? Thanks!