If you can stop payment, I would. His contract included lining - it was up to him to bid the job right, then do what he bid. Also, $5-6K sounds about 3-4 times too high for this - the entire furnace job should probably not have been more than about 1/2 to 2/3 that much. If the existing vent is too small to line then it probably meets code for dimensions, if not then he should have figured on opening up the area (tearing out existing liner ?) to make room for the duct. Alternatively, direct venting out the wall might be possible (depending on furnace model) which would typically cost about $500-800 including ducting and forced-evacuation fan (if needed) and safety interlock.
Personally, I think this whole setup stinks - makes me wonder how much you paid for the whole job versus what it should have cost.
You have a definite contract claim here - a pity you paid in full. I would get the local city/county building HVAC inspector to come out and flunk it in writing. Provide him with the above, where the contractor said it did not meet code. You then have good evidence of inadequate performance - send him (maybe wth help from an attorney) with the failed inspection, and that you have no faith he can remedy it, so you are demanding X dollars back to have it done right by another contractor. You will need a firm quote from another contractor (preferably two) for what it will take to fix it, add your attorneys fees, and that is your X amount. If he refuses you cold have Angie's List try to mediate it, but that would probably result in him wanting to "fix" it himself, which is not what you want. I would then complain to the state licensing board about his failure to adhere to the contract, and knowlingly installing a not-to-code system. If they take action and find he violated code, that combined with the inspector's flunking it and written quotes to bring up to code from other contractors will provide your attorney with ammunition to sue for professional malfeasance. Of course, if he was not licensed, then that strengthens your case even more, bringing in civil fraud and the possibility of multiple punitive damages as well.If you can stop paymetn, I woul