Actually, both transitions are equally annoying to me.
I think there is some natural talent involved with acting as well---some people just can't act. I know I could never do it, and I bet lots of other people can't either. It's one of the arts, just like music, visual arts, etc.
Singers--> Actors is sometimes worse in my opinion, just because it ruins the entire production. At least with singing, you can ignore that one person's album, but if some singer gets promoted into a juicy role of a movie that originally had a good plot, now you can't watch it due to their shitty acting whereas the movie itself might not have been so bad.