Apparatus and method for transmission and reception in multiple input multiple output system with relay station

A multiple antenna system for using a relay station with multiple antennas is provided. The system includes a base station for determining encoded data processing procedure based on first and second channel state information, determining a relaying procedure of the relay station, and sending the relaying procedure to the relay station based on the first and second channel state information; wherein the encoded data is terminating to a mobile station; the relay station for receiving data and the relay procedure which the base station sent, converting and sending the data according to the relaying procedure, and sending the first channel state information to the base station as a measured state of channel which receives the data; and the mobile station for receiving the data which the relay station sent, and sending the second channel state information to the relay station as a measured state of channel which receives the data.