My brief research on these hasn't revealed a lot of overwhelming appreciation for the current choices. In my experience, while formations do have some advantages, the current variety all have one very serious drawback. Whenever the fleet commander changes direction, all participating ships on the outside perimeter will be forced to travel ever larger distances as the fleet size increases. Often, even minor changes in fleet direction will result in several fleet members having to COMPLETELY TURN AROUND to reposition themselves in accordance with the formation. This, in turn, has very lethal consequences as these ships will lose a substantial amount of their firing arcs. More critically for destroyers, their main batteries will no longer be facing in the fleet's cardinal direction (presumably the direction of incoming threats), robbing the fleet of a significant portion of its firepower and creating huge delays in allowing said destroyers to recover their battle stance.
Thinking back to the original Homeworld RTS (not the recent remaster), I'm reminded of two formations that were overwhelming effective:
Loose Formation - Allow the formation members to simply position themselves anywhere, however they need to, so long as they remain within some predetermined distance of the "formation center". Because there would be no actual rigid formation, there would be no constant repositioning. This would allow each individual ship in the fleet to turn IN-PLACE whenever the fleet changes direction, thereby eliminating the need to constantly reverse direction and become over exposed to harm. Ships are also free to move about to "rebalance" their distribution. But there would be no rigid formation itself. This would basically allow players to take advantage of the Attack/Defend/Intercept/Bombard/etc. fleet directives without the disadvantages of the existing fleet formations.
Sphere Formation - This would be similar to the existing Circle formation but extend into three dimensions instead of just two. As the current circle formation is projected onto the vertical (YZ / YX) planes, it causes the entire formation to reposition in order to rotate with the fleet direction. In stark contrast, a full sphere formation would allow each formation member to turn IN-PLACE as with the loose formation above. When the commander turns slightly left, everyone else simply stays where they are and turns slightly left (instead of firing up travel drives to get to a new position). Unlike the current circle formation, the sphere formation would be able to turn on a dime without compromising formation coverage.
In both of the above formations, the formation center would be the commander when not actively in battle. Ships tasked with defending would always remain in their screening positions around the commander. Actively attacking ships could have their formations optionally center upon the target. In the case of the sphere formation, this would attempt to surround the target on all sides. I propose this should only be an option simply to avoid the complexity and risk of trying to surround targets. A basic model would be to simply attack from the current position and only reposition as needed to fire on the target. Distances to the "fleet center" (i.e. formation radius) could be assigned as a function of the largest fleet member's ship mass, so that larger ships would have greater berth to maneuver whereas fleets comprised of only fighter escorts would be allowed to have tighter formations.
Lastly, in case both of the above are completely undoable, could we at least get an alternative Circle formation that is projected onto the horizontal (XZ) plane instead of vertical and which would allow its constituents to turn In-PLACE instead of completely repositioning?