Anyway, onward into the realm of imagination
I've looked through Beastmasters in other IP's and WoW's Beastmaster isn't typical. For example, in DOTA 2, they are a melee strength hero. In Rift, they are plate wearing warriors with spirit companions. In D&D, they are medium armored rangers with animal Warders. In each of these incarnations, they are damage dealers, often with combat controlling abilities, but usually not tanks. They are also nearly always melee, often with axes specifically. My point is, there is room for variety. I suggest then a hybrid ranged/melee tank. Call it Beastmaster-Guardian (BMG), since I can't think of anything else. Sorry.
Ranged Tanking :
In my opinion, Hunter's pets make fantastic tanks already, so give Beastmaster Hunters a couple more tanking tools and the ability to select tank on the queue. To be viable for the average player, Hunter pets would need a less buggy Move To command, the same 500% threat buff as other tanks, and a buff to pet health. Hunter pets already have reasonable AoE threat, uncritability, +50% healing taken, and 90% reduced AoE damage.
For our new spec, BMG, let's keep all Hunter abilities that are non-specific to specializations, i.e. Serpent Sting, Arcane Shot, Steady Shot, ect. All ranged abilities can be used to good effect in Aspect of the Hawk and with a ranged weapon equipped, so this range-tanking would be effectively similar to how Hunters pet-tank already, just with a few weaknesses addressed
For better pet threat, make Misdirection to the Hunter's pet a permanent buff, and Distracting Shot apply automatically to the pet, both while in Aspect of the Hawk.
Add a new, spec specific ability called Raptor Shot in Aspect of the Hawk, changing to Raptor Strike in Aspect of the Beast (detailed later). Raptor Shot applies it's buff to the Hunter's pet, while Raptor Strike applies it's buff to the Hunter.
This would be the old Raptor Strike, with the Glyph reducing damage taken baked in and maybe lasting a bit longer, costing high focus, and unable to be parried (making the 7.5 hit and 7.5 expertise caps functional). Mastery increases the amount of damage reduced. I think an interesting way to make Raptor Shot/Strike even more unique from similar tank abilities would be to make it last longer when it hits critically.
I also suggest reviving Deflection, as an active 30 second cooldown, and making it increase parry and dodge significantly, the amount increased by attack power.
Add Serpent Spread to Multi-Shot from the Survival spec, but without the initial bonus damage. Without Survival mastery boosting this ability, I don't think it would be overpowered.
Finally, give pets Vengeance while in Aspect of the Hawk, 100% of the Hunter's stamina (the much needed health buff) and 100% of the Hunter's avoidance (making Deflection apply to the pet as well), along with a bonus 7.5% expertise to ensure that pets do not miss from the front. This solidifies pet's threat and survivability.
Revive Aspect of the Beast as "Increases melee attack power. Increases threat generation. Pets in Follow mode attack enemies in melee range, preferring the Hunter's main target, and their threat is transferred to the Hunter. The Hunter cannot be critically hit, and gains Vengeance." Have the Hunter equip a melee weapon (staves, polearms, or dual wielding) and tandem-tank with their pet on follow. One problem that immediately arises in melee is that ranged abilities require a ranged weapon. Simply remove that requirement for BMG. As current Beastmaster Hunters have an axe swinging animation for Kill Command, add a gun or pistol firing animation for BMG Hunters with a melee weapon equipped.
In Aspect of the Beast, certain Shots are replaced with melee abilities:
Raptor Shot - Raptor Strike
Serpent Sting - Lacerate, applies a bleed debuff (overwrites Serpent Sting, so no double dipping)
Arcane Shot - Counter Attack, still procced on parry, but with no cooldown, low focus cost.
Steady Shot - Mongoose Bite, generates focus, no cooldown.
Multi-Shot - Multi-Strike, spreads Lacerate.
A basic single target priority emerges of: Maintain Raptor Strike buff>Counterattack>Maintain Lacerate>Use Mongoose Bite to build focus. A multi-target rotation might be: Maintain Raptor Strike buff>Lacerate once and Multi-Strike to get debuffs on all targets>Counterattack>Multi-Strike to dump focus>Mongoose Bite to build focus.
We have a nearly full tanking toolkit. Counter Shot is our interrupt. Distracting Shot is our taunt. Deterrence is our main cooldown. We have a great variety of pets with buffs and debuffs to choose from, and I imagine Weakened Blows being the highest priority, if not otherwise present. We have self healing talents at level 45. We have focus increasing talents at level 60 that increase survivability when spent on Raptor Strike.
It still needs some more survivability, though. Reduce Deterrence to 1 minute to bring it in line with other tanks, while adding in a longer, stronger cooldown. I suggest Thick Hide, yet another throwback, but now as a 50% reduced damage taken, 3 minute cooldown. Self healing talents at level 45 would need increased potency for BMG, but in concept work well. Between these three and Deflection, managing damage incoming should be mostly fine.
Baseline survivability relative to shield tank's block still needs to be addressed. I suggest a passive ability, Primal Bond, causing damage done to either the Hunter or their pet to be split equally between them, effectively creating a shared health pool, and a percentage of healing done to either the Hunter or their pet applied to the other. This doesn't actually reduce damage at all, but significantly reduces the "spikiness" of damage. As a nice bonus, this would make Mend Pet heal the Hunter as well. If Mend pet were changed so that it scaled with haste, could crit, and maybe could improve based on AP and Vengeance, it would become a significant source of healing, thus gaining the survivability balance.
This also means BMG Hunters would have a significantly higher health pool than any other tank. For reference, my Paladin tank, fully gemmed for stamina and with two upgraded stamina trinkets, has about 743k health before buffs and a 7522 score on WoW Heroes. A Hunter similarly scored at 7522 has 458k health. If Hunter pets ended up with matching health pools to their Hunter, that would represent a 916k health pool, before buffs, without stamina gems or trinkets. That combined with a pet's 50% healing taken means that, thought they would likely take more damage overall, they would be easier to heal, especially if HoT's could double-dip. Having twice as many HoT's running would greatly improve throughput. Having significantly more total health would mean fewer expensive heals necessary throughout a fight, actually reducing the total mana needed to heal the BMG tank.
Stat priority is anyone's guess, and would just depend on how it all maths out. Agility along with Vengeance buffs Deflection and Mend Pet, so it would remain relevant, but not necessarily as powerful as secondary stats. Haste and crit both increase uptime of RS. Mastery increases the potency of RS. Dodge and parry are largely irrelevant, since they don't appear on agility mail, and would likely not be reforged into, similar to how Monks and Druids also don't reforge into avoidance.
Wait, Lacerate doesn't suck anymore?
In closing I again refer to the main principle of bringing something new to the table. What we have is a BMG Hunter that can pet-tank at range in Aspect of the Hawk, or tandem-tank with their pet in melee in Aspect of the Beast, depending on circumstances. I would think that most add tanking would be done in Hawk, while boss tanking, or tanking adds needing precise positioning would be done in Beast. We have a mechanic softening incoming damage in the form of shared health pool, which is not strictly new (Soul Link, a Warlock talent), but is new to real tanking. We have an agility mail wearing tank.
Next up, Battle Mage!