Some problems using somewhat precise fractional numbers with lots of decimal places (6-7, which AFAIK arent usually enough to cause problems with ~8 byte approximating representations) have given me incorrect answers. I suspect this is because of the usage of a limited precision floating-point representation for fractional numbers. If this calculator is using a limited precision floating point representation for numbers, then use an arbitrarily precise representation for fractional numbers. One potential representation that you could use could be a fraction struct. Scheme and Racket interpreters use that representation for arbitrarily precise fractional numbers. It works well. Either that, or something has gone wrong in parsing the mathematical expressions that gave me an incorrect answer. (Or perhaps me and my stats teacher both need glasses).
Grainysanddollar about MyScript Calculator - Handwriting calculator, v1.3.2