This is a really good use case. I thought of doing this myself at some point. For this, I think you may need to make a separate ingress resource that scoops up exactly that path, so that path comes with the annotation. Since they use the same group, all the ingress resources will be collected to make a combined rule set.
For alternatives, if you use ACL feature (which requires enterprise license), you have have logins with non admin accounts. The admins have to be members of the group Guardians.
You can also test the pattern out with ingress-nginx for a compare point, in case there’s something funky with ALB.