Yes, I would say that it would be cruel. If it were a new kitten that had never been outside before, then it might be a bit different. However, as we are talking about an adult cat who has been allowed outside for the whole main part of his life, it would not be fair on him.
What do you mean by "given me reason to believe she might hurt my cat"? Has she done anything wrong? As always, I would strongly advise that, if she has done something illegal (cruelty to animals counts here as well), you call the police. If she is being threatening, this, too, could be a case for the people who dress in uniforms and drive police cars to deal with.
Back to the subject of your cat being allowed outside, I would advise that you make sure that the fence is secure so that he cannot get out and just let him go out in your garden/yard.
Best of luck.