How do you graciously get out of hanging out with your husband's friends?

My husband's friends are absolute losers. None of them have anything going for them. some of them still live at home. (we are all over 35). And he has a couple of them that are on welfare and "work the system" to get benefits, etc. I have gotten to the point where I want to just let him go do things with them by himself, but he always wants me to go with him and sometimes gets upset when I don't want to go. Should I just continue to put up with them for his sake?