I thought it was obvious why the Operative flipped. Remember he 'kills and never asks why'. He explains to Mal at one point what his goals are...a world without sin. Something he, as a murderer, cannot be a part of. But he believes everything he does is for the greater good. Mal shows him what a world without sin would look like, and what the 'greater good' costs. It's a *dead* world.
The Operative has done what he knows to be terrible things, things which completely damn him. And the ends haven't justified the means. And he's not turning against the Alliance, he's simply walking away. There's actually a slightly longer ending there which was cut, where the Operative asks Mal how you keep going when you've lost everything. Mal is terse with him, Operative says his shit, and as he walks away Mal says under his breath, 'What a whiner'. I can see why they cut that, but I think it shows more of the Operative's character. Without a purpose, he's nothing, and he has no new purpose.
Doesn't mean all their secret agents suddenly will turn. Doesn't mean all the people will either. The woman giving the report on what happened even justifies it, even as she's seen what it did. "We meant it for the best..." Even the Operative knows that while eventually the govt. might fall due to this it won't happen overnight, if at all. It's not even going to generate enough heat to guarantee the Tams' safety.
And if you've watched the series you'd realize Alliance control isn't so weak. Yes on the outer moons where the Serenity does a lot of work, their presence isn't as obvious...but even then they're there. Train Job has cars full of Alliance soliders. During the series they visit 'core' worlds (basically imagine big cities) which are the heart of the Alliance.
The analogy would be simply, the core worlds are the big 'cities'. The outer moons are the back wood jerk water towns. Sometimes they're run by corrupt officials and the 'federal' govt. isn't directly there...but the small town sheriff is. However, fuck around too much and the federal govt. will show up and step in. Same idea. Recently on the news there was an outcry over cops stealing shit from motorists in a small town in Texas, no charges filed but property was 'seized' all the time, including one old lady's life savings. Shit like that happens in our country unfortunately, and it's getting fixed now which is why we're hearing about it...but it went on a long time. It's probably still going on or other shit like it in various small towns. Doesn't mean that the United States has a tenious hold still on the south. ;D
As far as most of the browncoats itching for another fight, they're not. It's very similar to the South getting their asses completely kicked at the end.; Even if they wanted to, there's no resources for it. They were effectively smashed. Whedon came up with the show's concept after reading 'Killer Angels' and basically wanted to show people who were on the losing side of a conflict, in a pioneer-type society that had to live in the society they had fought against.
Interestingly both in the series and in the movie, Alliance people tend to assume Browncoats are still fighting the war, or want to. The Operative even makes reference to this about understanding how Mal must feel about the Alliance, and he says 'No, you really don't.' I personally think it's obvious they know it's a war that couldn't be won, for various reasons, but they simply want to be left alone to do their thing. Which essentially was what the war was about.
And yes, in a sequel I completely see them going back to doing jobs. That's what they do, and it's how they eat. At the end everything was going back to a state of balance on the ship. Zoe's speech is pretty much about herself rather than the ship. Kaylee and Simon are now together. Jayne is...Jayne. And it's hardly the first time he's had problems with the Capt. or crew so there won't be issues there. And Inara and Mal...who knows. Mal hates what she does for a living, he doesn't respect it, but he respects her and seems to be in love with her. Inara left when she realized she had feelings for him (she got jealous over something he did). So...who knows.
They had resolved their conflict with another set of agents before this (the Hands of Blue), and now the Operative. The Tams issues were always a side plot (sometimes direct plot). I doubt they'll be left alone. In a sequel I'd expect to see they're still keeping low, either with the govt. officially after them, though not so resource intensive as they have nothing super secret to keep hidden, or personal vendetta by one or more politicians who are very powerful.
I wouldn't *mind* seeing another civil war mind you. I just don't think it's realistic within the storylines/universe. I also wouldn't mind seeing major instability in the central govt. leading to a better situation. But not holding my breath there.
All opinions though, and if you liked the movie, you really have to see the series, a lot more becomes clear. I saw the movie first then the series myself.
Frankly I'd just be happy as could be for ANY sequel...any at all. Though I'm sad that they killed off some people.