Peopling your story

Some other site today talked about choosing the right name for characters. which reminded me of an issue I am constantly confronted with. Is it permissable for me a white, middle-aged woman of European descent to make my characters black, ethnic, male, young ,etc?
Well maybe some of these things, but what about portraying African-Americans? Am I claiming ownership of experiences I haven't had when I write about a black man? Is it offensive? Should I even think I might be successful at it.
I wrote one story about a black man and his grandson a few years ago for a journal that deals with disabilities. The man was deaf in the story. Now I had no problem with thinking I could show what it was like to be deaf but I chickened out on his being African-American and took out all the hints. I probably know more about being black than deaf but didn't concern myself with offending the deaf community who would be more likely to read the story in the journal it was in. Should I stick with portraying a white world? Isn't that more offensive in the end?