This research aimed to identify the factors in peer groups which positively and negatively influence the students’ anxiety in learning English and determine which peer groups have more effect to the students’ anxiety in terms of grouping division by the teacher or by the students. The researcher applied descriptive qualitative research method. The data collection was done through observation and interview. The participants of the research were the second grade students of SMAN 2 Parepare. They were selected by using purposive sampling referring to the English teachers who provide their classrooms in grouping teaching method and also referring to the students who were involved in peer groups. The result of data analysis showed that some factors which positively influence the students’ anxiety in learning English consist of communication skill development, sharing diverse assumption, and collaborative learning. Meanwhile, students’ self-perception, low language proficiency, and peer rejection are factors which negatively influence their anxiety. Besides, competitiveness becomes one factor in peer group which positively and also negatively influence the students’ anxiety in learning English. Furthermore, grouping division by the teacher which created heterogeneous grouping gave better impact to the students’ anxiety in learning than by the students which created homogeneous grouping. Therefore, it is suggested to the English teacher to know how to create a good condition in teaching and learning environment among the students in order that they can do some methods or teaching strategies that provide students to work in a group. They can be more concerned with some factors that cause students’ anxiety and help them to reduce their anxiety in EFL learning.