You can only do one thing: open the door and let him in for good (subject to the practicalities of that suggestion)!
This is because you have created a relationship in which the cat has become dependent on you for sustenance. This has developed into a human-cat relationship as is found in millions of homes. The cat wants to move on to the next stage: go inside where it is warm and secure.
I’d steel yourself and take him in. You obviously like him and are fond of him. Why not take the extra step and make the relationship a classic indoor/outdoor one?
The problem is an interesting one. An act of kindness has created a burden or more precisely a responsibility. I guess this was not foreseen by the lady. These things do happen. When a volunteer feeds feral cats as part of a TNR program that person also creates a responsibility for the cats.
This leads to an emotional demand on the person. An emotional connection is created between cats and volunteer and it is hard to shake off. Someone said that if you feed a cat regularly you own the cat.
I don’t agree with that exactly but what follows is something akin to cat ownership. It is certainly cat caretaking or cat guardianship which arrives by default.
If a person kicks of a relationship with a stray cat they should be prepared to accept the possibility of it developing into a full-blown cat owner situation.
However, I do understand the difficulties that that might create. Not everyone has the time, funds or commitment to informally adopt a stray cat.
P.S. Make sure the cat does not have an owner by checking for a microchip at the veterinarian’s.