Tanking MethodThe first issue to address, I think, should be a way to make Battle Mages a new way to tank. A unique method would be to have damage taken split between mana and health. That would make an effective 300k health bubble passive to the spec, and make Mage's current mana regeneration abilities relevant to damage mitigation. Damage taken to mana would be replaced by Evocation (also a health return with it's glyph), Mana Gem (which also has a useful glyph), and level 90 talents (Invocation, Rune of Power, Incanter's Ward.
Finally, add a reverse of Warlock's Life Tap (a heal by any other name), call it Mana Tap, and the health bar is also taken care of by this active mitigation model. Damage comes in, split between health and mana>Mana is regenerated>Mana Tap brings health up>Mana is regenerated>Rinse and repeat. A unique Mastery could be to increase mana regeneration. It could be a boost to passive mana regeneration and/or a boost to mana regeneration abilities.
The armor would be Intellect plate. Since other plate classes start with mail armor and don't use plate until level 40, Battle Mage should probably do the same. The weapons would be with 1 handed and a shield, or 1 handed and a staff. Equipping a shield would boost armor (33% of my Pally's armor comes from my shield) and enable blocking, while equipping a staff would instead boost Spell Power (one of my healers gets about 39% of his SP from his staff) and enable Nether Attunement. This make a choice between extra physical damage reduction or extra mana regeneration.
CooldownsUsing existing Mage abilities in the game seems like a good place to see what else they could do to mitigate damage.
The first major cooldown would obviously be Alter Time. It would work for any type of damage, magic or physical, big hits or death by a thousand cuts. Any "bad" happening within a 6 second time frame can be completely negated. Also, defensive cooldowns can be doubled-up for more survivability just as Mages currently double up offensive cooldowns for more DPS.
The other defensive cooldowns already available are level 30 talents (Temporal Shield, Flameglow, Ice Barrier), level 60 talents (Greater Invisibility, Cauterize, Cold Snap), and Ice Block. Invisibility and Ice Block could be gimmicked a la Pally's Hand of Protection. i.e. taunt right before using the ability and the enemy will fixate for 3 seconds. They could also be modified to not drop aggro for the spec specifically.
Currently, other tank's Symbiosis abilities are Might of Ursoc for DK's, Survival Instincts for Monks, Barkskin for Pallys, and Savage Defense for Warriors. Guardian Druid's remaining major cooldown would be Frenzied Regeneration, so make that cost X% base mana and we have another cooldown.
For passive damage reduction there's Molten Armor, and another version which is in common use by NPC's. There's also Mage Armor, and Glyph of Armors. The glyph makes Armor swapping feasible, but not spammable and increases the effects. Add a passive to the spec that make all Armors reduce the chance to be critically hit by 6% and increase threat generation. I could see the decision between Molten and Mage armor being an interesting play choice based on a fight-to-fight basis, or even swapped during different phases. Molten would prevent more damage, but Mage would enable more health/mana replacement.
RotationA rotation could be easily built using existing Mage abilities, and I think the Arcane spec is an appropriate theme to copy. An Arcane Mage is already concerned with managing mana levels, and it lends itself well to the Arcane Warrior/Spellsword style. I don't think it would be too difficult to manage Arcane Charges while also managing health/mana levels. Replace Arcane Barrage with Mana Tap and make it cost Arcane Charges. That way the rotation becomes relevant to survivability.
My first concern is that offensive abilities cost mana, and mana is part of this kind of Battle Mage's defense. Using Mana to deal damage costs effective health. I think this is fine, as mana regeneration is already a "push-pull" concept for the spec.
My second concern is all the casting Mages do. Simply copy Warlock's Dark Apotheosis concept and make it passive for the spec. All abilities with a cast time get an instant version. Also add auto attacks to further increase threat generation.
StatsIntellect plate has a lot of Spirit on it, so making Spirit convert to Hit would fall in line with caster hybrids. To change things up a bit, keep all abilities as spells, not melee abilities. That way they would not be able to be parried, resulting in a standard 15% spell hit cap for all special abilities.
Mastery would probably be very powerful at first, but less so as stat levels inflate, so it would probably have effective diminishing returns. Once a Battle Mage has enough Mastery to not run out of mana, other stats would become more relevant.
Haste reduces the global cooldown as well as increasing mana regeneration when using a staff, so up to 50% Haste would remain very powerful for building Arcane Charges.
Crit is so far irrelevant, so have special attacks that crit grant a free Mana Tap (or a chance of one), and to avoid wasting this, give these free Mana Taps 2 charges like Arcane Missiles.
Spell Power could be made relevant by increasing the amount healed by Mana Tap.
How a stat priority would work out would just depend on the math. I think 15% Hit would be mandatory to ensure Arcane Charge generation. After that, I'd see an approximate level of Mastery being needed, then either Haste or Crit stacked, whichever theorycrafts better.