Can a male and female teacher talk?

Okay, here's the deal... during my first week of school, second day of school with the kids, I had a male teacher enter my room during my planning period to discuss one of the inclusion students that he was working with in my class. Because we were two teachers, talking about another student, we both deemed it appropriate to close the door. Please, keep in mind that my door, to my knowledge, at that time, could not lock from the inside; also, all of the doors have windows for people in the hallway to see into the classrooms. Anyway, the teacher sat down at a students desk in front of me, and I continued sitting at my desk. About 5 minutes later, there is a knock at the door, and my principal and some big-shot from the main office drop in to check out my classroom set-up. I kindly let them in, talk about my teaching strategies, and they move on. Approxiamtely 2 minutes later, the Asst. Principal comes to my door and asks to talk to the male teacher I am having a discussion with. Thus, I go back to sitting alone in my class, grading papers. Low and Behold, my Principal comes knocking around my door, once again - this time, alone, and serious. "What were the two of you doing in here?," she asks me. "Talking," I respond. "What were you talking about?" I was a bit startled, but I calmly explained to her that we were discussing a student that the two of us were working with that day. "Well," she says to me, "When a person walks by in that hallway and sees a young couple locked up together in the classroom, things do not look good. There are rumors starting , now, about your relationship." Stunned, but not wanting to put myself in a worser situation, I ask what the appropriate protocol is for two teachers who want to have a confidential discussion about the well-being of their mutual student. She responds, "Talk in the hallway, and make it brief."
Two weeks later, and this situation is still being brought up to me. Teachers in other schools, the Asst. Super-intendent, and others have brought this rumorto my attention. Now, the male teacher, who was working in my classroom for two periods a day with inclusion students, has been given a totally new schedule - never to have a class with me.
Is this weird to anyone else? I told the story to some other corps members, and at first they were shocked, but when I mentioned that the male teacher also happened to be black (I'm white), they suddenly became less surprised.
Who was unprofessional in this situation? Myself and the other teacher? My principal? The administration? Everyone? No one? I'm still just kinda baffled.