Legally, without a court order you cannot deny him access to the child. Does he have any arrests for alcohol related issues? Drunk and Disorderly? Public Intoxication? DUI?
If the answer is NO, then unfortunately it's your word against his. It sucks, but you can't stop him from seeing her. HOWEVER if he shows up smelling of alcohol, you are within your rights to refuse to allow him to take her.
(There was another post around somewhere from someone whose ex was sending their alcoholic father to pick up the kids...however as that poster was told, as long as he shows up sober, you can't deny access without a court order).
Perhaps you should try to get his family on board your side? Are things amicable between you and them? Perhaps a phone call or a letter to them indicating that your only concern is his drinking, given what you have first hand knowledge of, that any legal issues are between him and you , and that you realize that and will endeavor to promote their relationship with the child, no matter what happens with her father.