Am I paranoid? Or do I have reason to doubt him?

Since he is hiding information about a woman he has known for two years, there is every reason to be paranoid. While it is possible that he is not having a full blown affair, the whole 'calls drunk at night' hints to your husband and Ashleigh having some relationship ( maybe past or present).