Side-channel risks of Intel SGX have recently attracted great attention. Under the spotlight is the newly discovered page-fault attack, in which an OS-level adversary induces page faults to observe the page-level access patterns of a protected process running in an SGX enclave. With almost all proposed defense focusing on this attack, little is known about whether such e orts indeed raise the bar for the adversary, whether a simple variation of the attack renders all protection ine ective, not to mention an in-depth understanding of other attack surfaces in the SGX system. In the paper, we report the rst step toward systematic analyses of side-channel threats that SGX faces, focusing on the risks associated with its memory management. Our research identi es 8 potential attack vectors, ranging from TLB to DRAM modules. More importantly, we highlight the common misunderstandings about SGX memory side channels, demonstrating that high frequent AEXs can be avoided when recovering EdDSA secret key through a new page channel and ne-grained monitoring of enclave programs (at the level of 64B) can be done through combining both cache and cross-enclave DRAM channels. Our ndings reveal the gap between the ongoing security research on SGX and its side-channel weaknesses, redene the side-channel threat model for secure enclaves, and can provoke a discussion on when to use such a system and how to use it securely.