A lot of what you can and can not do depends on the laws in your state. If you can prove he is being negligent and/or not doing the job properly you have grounds to fire him without further cause. Did he give a reason for the job going over the estimated timeframe? It sounds as though he is very disrespectful to you and your home. From a Contractor's perspective though there may be things that came up which presented additional problems (plumbing, structural, etc.) which extended the scope of the work. If this is the case he should have spoken to you about it. Either way, it is inexcusable to not put forth a full effort to get the job done in a timely manner. A good contractor will not be able to spend all day at one job while meeting with other customers, writing estimates, and handling the office duties of his business (book keeping and such) so you can't expect him to spend all day every day in your home for three weeks straight. However, from what you have described he has little interest in getting the job done. We've all had jobs "sour" on us where nothing seems to go right and problems abound, pushing us way over our intended timeline and budget. The difference is the communication necessary with the customer and explaining what's going on.
Look at your contract, make sure he has any applicable permits pulled, and have someone inspect his work thus far. If it is not up to snuff or he hasn't followed the law as it applies to your remodel fire him. Document everything. Having your local municipality on your side as Kenny mentioned will help. Most are quite cooperative with homeowners (often more so than with contractors), especially when you explain how you were taken for a ride and if he did not pull the permits if there are any required in your area.
Todd Shell
Todd's Home Services
www.thomeservices.com