Anyways finally had some time to take another crack at this. I notice the CORs issues were due to Referrer-Policy: no-referrer-when-downgrade.
I figured it was because I was connecting from https://play.dgraph.io/ to http://xxx.xxx.xx.xxx:8000, meaning, it went from HTTPS to HTTP, so the referrer header was lost due to the policy.
To fix this, I went ahead and set up my server with a domain and SSL cert. After doing so, connecting from https://play.dgraph.io/ to https://my-domain.com/dgraph totally works as expected.
And also consequently, connecting from a GraphQL client (like Altair) works as well 