Sorry for the necropost, but it turns out the reason for the strange behavior at 10 is pretty simple: The heuristic weight is silently capped at 9. I don't know why. Additionally, the costs are capped at 254, and none of the three values can be set below 1.