Well, theoretically everything in Spelljammer generates its own gravity and can thus hold an air envelope (if passing through or starting from an atmosphere) - even a halfling (or smaller)!
The wording in the original description, noting specifically "In atmosphere, ..." implies to me that it has to be in an actual atmosphere (not air envelope) to do so, and those are different things in Spelljammer. (Though I do see your point.)
With that said, I'm not picky, so I'm okay with how you want to lay it out. (Always flames, and if you want, some type of area. Though if it makes it too complicated, I'm okay with leaving the 'area' damage out. Having an area of flame makes it a bit more complicated when adjudicating a spelljamming ship fight, which doesn't need any more complications!)
Yes, I agree the wording suggests that the original intent was that a gammaroid can only flame sheath in a planetary atmosphere, it's mainly that the explanation for it doesn't make much sense to me.
The text says it spins so fast that atmospheric friction creates a sheath of flame around it. So it creates flame like a meteor burning through an atmosphere does.
To generate that sort of friction the outer rim of its shell would have to be travelling at high atmospheric re-entry speeds - tens of thousands of miles per hour. A typical meteor hits the earth at about 18 miles per second, or about 65,000 mph. A 2,500 foot diameter gammaroid would have to spin at about 725 revolutions per minute for its shell edge to reach that speed.
That's
vastly faster than tactical spelljammer speeds. A gammaroid's 9 SR works out as either 1534 mph or 153 mph depending on whether we're talking 500 yard or 150 foot tactical hexes.
If it can spin like that in a planetary atmosphere, why can't it do it in an air envelope? The air pressure's similar or else spelljammer travellers wouldn't be able to breathe. I can see two possibilities.
Firstly, if the gammaroid's air envelope rotates in sync with the gammaroid, it wouldn't rub against its shell and generate heat. The problem with that is it means the gammaroid is spinning in the middle of a hypersonic hurricane. Any objects that impact the air envelope may not take damage from heat, but they certainly would from the winds travelling at tens of thousands of miles per hour.
Secondly, the gammaroid may only be able to spin that fast when its near a strong gravity source like a planet. That'd fit the facts better, but how does it do it? Spelljamming doesn't work at anything over tactical speeds when you're close to a planet or any other body - that's why a vessel can't use "interplanetary" spelljammer speeds until they in clear space.
Frankly, it'd makes better sense if it generated the flame sheath by shooting fire out of the leg-holes in its shell like a certain other giant turtle...
I don't mind it being unable to use its flame sheath in airless environments, but I think it should be able to do so when it has an air envelope.
How's this for a draft:
Flame Sheath (Ex): A gammaroid using Spinning Flight can envelop itself in a fireball of superheated air as a free
? action. This Flame Sheath has a diameter of
X feet and does
Y fire damage per round to everything within it. The Flame Sheath is centered on the gammaroid and moves as the gammaroid moves. A gammaroid can maintain a Flame Sheath for as long as it wishes, but the Flame Sheath immediately ends if the gammaroid stops using Spinning Flight.
A gammaroid cannot create a Flame Sheath when surrounded by vacuum, since the heat and flames will instantly disperse into an airless void. This is not usually a problem for a gammaroid travelling in space, since their own gravity field carries an air envelope along with the creature.