Enmity will be a problem. BST pets inherently have ZERO good tools for generating Enmity.
Its almost entirely Damage Dealt vs Damage taken.
All enmity tools are on the master. Its not hard to have bst pet tanking when solo, and when master is engaged.
The primary tool ofcourse is snarl. But the master cannot stay out of range to use this. If you are paying attention, and skilled enough, you can run in and out of range for most bosses. Using abilities or spells from your support job, and snarling the hate to your pet.
To do this you would need a good Master Enmity set.
ItemSet 356576
And Pet Enmity Set:
ItemSet 356577
Using /blu and spamming cheap enmity spells from a distance, then running in to snarl enmity to pet is probably most effective way to keep hate on pet.
I'm not sure how often to use the Pet Enmity set in this situation. If you put on Pet Enmity set, you are losing your pet damage reduction, and vice versa. I'm not certain if it helps, but I equip pet enmity set when using snarl.
I often use heyoka on bst as a hybrid tp/meva set to melee in when i want pet to tank and I don't want AoE magic damage/enfeebles. However, if you are trying to tank something BIG, you will be staying mostly in pet -dt gear (physical, magical or switching back and forth). Like Puppetmaster, this is one of the best ways to retain enmity: NOT taking damage. This may present an accuracy problem for generating hate depending on target.
Even though pet damage reduction caps at -87.5%, bst cannot achieve this outside abyssea.
BST can achieve -70% magic damage on pet with gear and separately 82% -PDT with some reduction of -mdt. This can be helped a bit with pet magic evasion bonus on taeon hands/feet or pet defense from acro armor in a few slots. These perfect sets are difficult to obtain. I'm about 2% away still and I've worked at it alot.
this does not include the 3% bonus from vorseals.
Its certainly more challenging than using a pup tank, but I think it could be feasible for some Omen bosses. Rangers would steal hate in a heartbeat. but smn's would be fine. avatars might take hate, but you can easily let them die or dismiss them.