Mike, presumably this is a known good engine that has been starting OK in the past, rather than a new rebuild?
If a new rebuild, it could be that the engine is tight.
If a known good starter, and assuming your description is as accurate as it sounds, you're looking for losses in the electrical system. Check all the connections and try turning it over a bit, then carefully have a feel of the starter switch, all the connections and the starter itself to see if anything's getting hot. Wherever it's hottest there's likely to be a high resistance, i.e. poor wiring, bad connection etc. Clean up any connections, replace any suspect parts and try again. Don't forget to check all the earths - engine, chassis, battery etc.
Another point - has there been any rewiring? Low voltage electrics draw more current for a given power, so will need larger cabling; if it's been rewired has cabling to suit a 6V system been used?
Also don't assume the starter is good just because it's new; if it's NOS it might have corrosion internally due to condensation, if rebuilt it may not have been rebuilt properly.
Finally is the battery OK?
Edit - sorry for the echo, Arcot, we posted at the same time!