I think Don hit it on the head - if you got what you wanted, with him being a former home inspector, then you could pay him as a friend. If you were looking for a formal home inspection report under state law, then he failed to and could not deliver that, so I would tell him you are real disappointed to find out he is not licensed (presuming you found that out after the fact) so his "work" was of no value to you and under false pretenses, so you are not going to pay him. If he makes a fuss, make it clear a report to the state for practicing without a license will get him in a passle of trouble should shut him up.
Of course, the friendship will be shot, but likely is already if he is threatening to take you to court. If threatens to harrass your mother or actually does so, then timefor a police report - both for the harrassment and for fraudulent misrepresentation as an inspector. Of course, dealing with him through an attorney at this point would probably be best - getting a letter from an attorney will probably shut him up.
And of course, an appropriate review might be in order, in case he is doing "inspections" for other people too.