Depends on AWFUL lot on the building and fire codes in your area - you will have to get bids. Another significant factor is you say your furnace vents through PVC pipe through the (presumably outer) wall. Could there then still be usable double-wall galvanized vent duct to a roof hood from when the original furnace was installed ? If so, and the water heater will sit near the old one does (so no significant water pipe work needed other than hookup) and within about 6 feet of the vertical vent pipe, then very simple to install a gas heater. To run a few feet of gas pipe and new gas shutoff valve and flex tubing from the gas line near your furnace say $50-100 materials, another $100 or so for the vent ducting. Install same say $200-300, for a total $350-500 to prep for the installation - more likely toward the lower side if easy access and gas line, vent are close by.
Then add on standard replacement gas water heater purchase and installation (see most popular questions - this is usually in top 3) - roughly $1000-1500.
If there is no existing vent pipe to the roof, then IF your code allows direct wall venting, probably the same cost to direct vent outside through the wall. Plus maybe $200-400 to build a new platform and extend water and gas piping to place the water heater close to an outer wall if the inside vent duct run would be more than about 6 feet or so to the wall (duct has to slope down to the heater to drain any condensate, so long runs run out of headroom). You might also have to beef up your air inlet vent size to accomodate the code requirements for both the furnace and water heater air demand - maybe another $50 to enalrge opening and install larger air source vent or add another smaller one.
Once you get a couple of bids, your gas company should be able to help you with the calculations to see what your payback period would be. If it costs $500-1000 to change to gas water heater (assuming you need a new water heater soon anyway, so purchase and actual install of heater is about same for either type and not a factor in the payback calculation), unless you are in a very high electricity cost area, I would off the cuff guess the energy savings could pay off the change in 10 years or so. Of course, if your gas prioces are high or elecricity is very cheap, then might be it would never pay off.