If paint is actually coming off with just water and a rag, probably not able to clean it at all. However, what you see might be chalking - the surface (especially with latex) turns powdery (and usually whitish) and comes off with just a wipe (on rag, clothing, etc) but takes a good few years before it goes all the way through the paint, but does mean the paint is either well aged (typically 5-10 years) or is failing. Will hapen especially with interior paint used outdoors.
I guess you have a couple of choices - just spritzing the hand print area with a mixture of water with a bit of liquid dishwashing soap in it, let sit for a few minutes, then gently hose off might remove the worst of the stains and surficial dirt without taking off the paint, though unlikely to take them completely off without some scrubbing - which might remove the paint.
Otherwise, try a spot that will not show badly (maybe along the very edge of the front face with door partly open, on the edge strip where it is concealed behind the doorframe when door is closed) and try a bit of dishwashing soap or laundry detergent on a soft rag or sponge and wipe it well (without getting too aggressive) and see if it is just surficial chalking or the paint actually coming off. I have seen some metal and fiberglass doors come in from the factory to jobsites lose their paint just sitting a week or two out in the weather, so could just be a really cheap door with bad paint, too - in which case a total paint removal and repaint would be needed. Or it could be it was an aluminum door that someone painted themselves without proper surface prep, so the paint did not bond correctly.
Here is another prior comment on cleaning and painting which I made, which might also be of interest to you -