Inmates at Long Lartin Prison said some were "becoming Muslim" because it is a "bigger gang".
In the same prison a radical cleric Abu Qatada (Osama's right-hand man in Europe) was held last year befor being released on bail - which he went on to breach.
Similar concerns were raised in an inspection report in to Whitemoor Prison in Cambridgeshire last year, in which prison officers warned extremist Muslim prisoners are ganging up on others in an attempt to radicalise them.
At Long Lartin, in Worcestershire, gang culture is widespread.
One inmate said: "Yes, there is a gang culture here which is becoming an issue. A lot of people are becoming Muslim just because it's a bigger gang."
Another said: "There are issues with Muslim gangs wanting to overpower others."
One added: "Muslim gangs; if you have a problem with one, you have a problem with them all," while a fourth said: "People are converting to Islam for protection."
Phil wheatley, director general of the National Offender Management Service, said: "The Governor and senior management team are committed to ensuring that all prisoners feel safe, and the strong personal officer scheme at Long Lartin continues to be developed to offer support to prisoners.
"A new programme to improve engagement, led by the Muslim Chaplain, will develop support specifically for Muslim prisoners."