A 10 inch prop might need something like 650W of electrical power, depending on rpm and propeller pitch mostly. A decent 18V battery drill motor may have a maximum rated power like this, or a bit less. Its peak power is more, probably 750W or so and that is intermittent rated. It will get too hot and burn out eventually. The motor might be something like 12500 rpm with 18V and the rated current 36 amps. As your load is close to the expected maximum continuous current rating, you need to watch the temperature of both motor and batteries. Ni-cad batteries are usually rated at 45°C maximum. Motor case temperature maybe more, but it depends on what else it is associated with, so 50 or 60°C is likely. The windings will be much hotter.
The battery has to deliver 36A for 1/12h = 3Ah (ampere hours), which all seems resonable. A typical C size NiCad is 2.5 to 3.5Ah. NiMH batteries could be a bit higher capacity than this, but if they are old, capacity will have fallen off.
The ampere hour capacity is (average amps * time in hours). With drilling operation it is likely intermittent use, so your average amps was more like 4A for 0.75 hours. Certainly the peaks are higher.
You can try other combinations. The motors used in model planes are probably more efficient, like 80% instead of 70%. Most are intermittently rated, and have flight times something like your stated time of 5 minutes. They often run off 3 or 4 cell LiPo battery so 11 or 14V typically. Typical battery capacity is 2.5 to 5Ah. These batteries are much lighter for the same watt hours. SO are the motors. If you used a 5 LiPo battery the voltage is 18V approx. With less voltage the power required by the propeller drops quite dramatically, so this would give a longer run time too. So it all depends just what power you want from the propeller. There is quite a bit of juggling to get battery, motor, propeller and speed controller optimized. Generally the overall weight is going to be important too. There are on line calculators that help understanding like:
http://www.ecalc.ch/xcoptercalc.php?ecalc&lang=en
I think the drill motor is good for testing propellers, but it is only meaningful if you measure the voltage, current and rpm and thrust. Note this is static thrust, more like a quad-rotor or hovercraft or during takeoff. The thrust at flight speed is different. The easiest way to measure this current is a DC tong meter with full scale at least 50A, maybe 100A is better.
The current with no load is still a few amps, but I would not expect it to be anything like 36A, more like 5 to 10. Therefore this is unexplained and doesn't make sense, unless it was a lot of stop and start during the test, so the peak current of starting could be making a difference.