If 'bar fight' isn't giving any ideas, 'prison fight' is probably just as good.
Granted, some prison scenes are meant to be more comedic and less gritty. "Jailhouse Rock" from
Blues Brothers for example is a dance party, not a prison riot. Most of these guys are musicians and look it
This scene from the "Folsom Prison Blues" episode of
Supernatural is mostly comedic, but they definitely cast Lucas and Tiny to be big burly guys for effect. I would not describe either of these actors as 'elven', though.
https://www.youtube.com/watch?v=DX-CECPhg_g
This scene from
Shawshank Redemption is the opposite of comedic (implied prison rape). The guys aren't unusually big or burly, they're just moderately strong but able to gang up on a guy.
The 'danger' vibe is definitely there with the three gangbangers in
Training Day. One of those guys is pretty ripped (in an obvious abuse of steroids kinda way), but all three of them have a very definite 'do not mess with me' vibe going on. Warning for language and violence and discussion of rape.
Edward Norton certainly put on a lot of muscle mass for his role in
American History X. [This particular scene is fairly innocuous, but still needs a language warning.
] Worth pointing out that even bulked up, he's not a particularly big guy. Again, they certainly cast some big burly guys to fill out this prison population to make everyone else look more dangerous than the swastika-tattooed skinhead murderer protagonist.
Jet Li's prison escape in the incorrectly named
Romeo Must Die. At least one of the guys he takes on is huge, but for the most part he's just precise and clever.
So, yeah, I'm fine with the 'Bruce Lee' suggestion - a guy of moderate build with some really compact muscle mass who looks like he knows how to use it and isn't afraid of a fight. Just, you know, taller than actual Bruce Lee. Typical elven build, but ripped.
[Ugh, it's tough when my knowledge of films/actors isn't recent. The 'kid brother' in
American History X is now 41 years old, and he's the same actor who played young John Connor in
Terminator 2. ]