I remember being introduced to terms of reference (ToR) when I moved into the public sector. Quite simply, I exceeded the ToR for an activity and said something that someone felt was ‘beyond my brief’.
As a penalty, I had to drive across town for an early morning meeting to be told not to do it again. I made the promise and carried on. Obviously, a few weeks later it ended up being done in the way I’d framed it.
Don’t use ToR to restrict permission, but use them to make them the framework about which people can exceed.