So I have noticed when I use a Hole Callout annotation I can change the tolerancing in a drawing and it affects the model.

I had assumed the Hole Callout was a smart dimension or driven dimension.

Is it really a model item?

For example I added a basic tolerance to these depths...

...and was surprised to see that in my model!

(Ignore the blue cosmetic thread dim, they never come out right anyway.)