This research aims at investigating several feature sets such as acoustic, lexical, and discourse features, and classification algorithms for classifying spoken utterances based on the emotional state of the speaker. Besides applications in enabling natural human machine interfaces, the problem motivates development of novel classification algorithms that can operate with sparse data. Another aim is to find out the way to seamlessly combine the emotion recognition system with state-of-the-art speech recognition system, and the satisfactory dialog management of the automated call center system to better support human-computer interaction in that application.