Hi - in many ways I can resonate with what you're expressing here, because although I didn't walk away entirely, I did 'abandon' my elderly MIL. My circumstances are a little unusual, in that I'm widowed, and my MIL's surviving son is the USA, so really can't actually 'do anything'. Apart from my student son, there is absolutely 'no one else' on my husband's side of the family to 'do anything' except me.
My MIL was very, very independent - amazingly so - right up until three years ago, when, at 89, she phoned to say she couldn't cope any longer. Up till then she'd lived on her own (400 miles from me) looked after her finances so competently she actually ran a surplus from her state pension (!), and although yes, she no longer flew up and down when she came to stay, but rather I drove her up and down the few times of year she visited, she was still amazingly good for her age.
All that changed when she phoned me, and I went into 'overdrive' thinking, OK, time to move MIL south, find a flat for her near me, and for me to see more of her, check she's got food in the fridge, take her out to lunch sometimes, that sort of thing - just the way it always had been.
BUT, it was totally different - she was developing dementia and basiscally because of that she simply 'collapsed' on me. She wanted to live with me and for me to look after her 'like a child' really. And that, sadly, was what finished it for me. After nine hellish months I cracked, and I 'put her in a home'. It was 'cruel' in that she'd got used to being with me (I managed to get her back to Glasgow for meagre two week breaks once a month, and she hated it, and really couldn't cope on her own at all - like your mum, she rejected outside carers, wouldn't let them in the second time they called) (she just wanted ME), but I did it all the same.
If she hadn't gone into the home (near me, very nice - she came for sleepovers with me twice a week), then I think I would have simply taken her back to her flat in Glasgow, given her key to her (long-suffering but very kind) neighbour, and phoned SS to say there was an 89 y/o vulnerable woman with dementia who was acopic, and I wasn't on hand to do a single thing so over to them..... That was my 'desperation point'.
I say all this to show that you are not alone in your determination not to have your life 'taken over' by caring. BUT, that said, I do visit her - not as often as I 'should', and I do 'feel bad', and it's slightly more complicated as she is now, for financial reasons, in a (cheaper) care home near my 'holiday house' in the westcountry, so I tend to not visit her while I'm at home, then head west for a fortnight and visit her several times and take her out.
BUT again, unlike your situation, my MIL's dementia is now very advanced, and she truly 'forgets I exist' when she does not see me, and this has made my guilt levels drop significantly. I tell myself that she does not understand the passage of time any longer, so is happy to see me, happy for me to take her out and about when I visit (we go for drives and a cream tea - perfectly pleasant, and I chat away to her, and it's all totally doable.)
BUT, again, I had a perfectly good relationship with her all my life, so there was never any tension between us at all. I sense that this is not so with your own mother? (What has happened to your father, by the way?). You say your mum is unpleasant to you - is this only because she wants to come home (and be looked after by you?) or was there always tension between you?
I must say, I never, personally, felt that I was NOT a 'caring person' but since 'inheriting' my MIL, I've realised that actually I HATE having 'dependent' people 'collapsing' on me - and for me, I know this goes back to being raised with a mum who had MH issues. I guess I feel I've done my 'looking after another adult' and don't want to do it again??
One thing I think is for sure - when we are faced with 'caring' it can bring out all sorts of things deep inside us that are not easy to resolve. Sometimes not even possible at all.