There IS a difference between attraction preferences and discrimination. You say "I only date people who are not fat." What if someone said "I only date white people, it's just what I'm attracted to." Would you still see that as a valid preference?
We all have attributes that are attractive to us (for example, I am often drawn to guys with beards, curly hair, or long eyelashes). But I don't say, I only date guys who can grow hair on their face. That's because it's just a preference, and there are always exceptions, because it's not a RULE.
Though the title says you're not fatphobic, there are several places in your writing where you indicate otherwise - for example, your comment about being "crushed" by someone in the bedroom. That's... not how it works? How do you think fat people have sex? No one is getting crushed, there are tons of positions and people don't just flop their entire weight down on top of other people, generally speaking.
The idea that you were in a relationship with someone who you WERE attracted to, but suddenly when he talked about being fat in the past and talked about his body in an honest way, you got unattracted to him? That IS fatphobia.