User association is crucial for optimizing the performance and utility of wireless networks, enhancing key aspects such as load balancing, spectrum efficiency, energy efficiency, and overall network performance. In this paper we tackle the user association challenge in wireless networks, particularly in resource-constrained connectivity scenarios. Our proposed approach, GROWTh (Graph Representation of Wireless systems Throughput fair), introduces a graph-based reinforcement learning framework that optimizes resource utilization through a fully decentralized algorithm. We validate GROWTh across diverse scenarios, including a 5 G deployment in densely populated areas characterized by high user density and traffic load, where it demonstrates significant improvements in various performance metrics. Notably, GROWTh achieves a substantial increase in system utility compared to traditional methods while simultaneously reducing user rejection rates. These findings highlight the effectiveness of GROWTh in managing user association in high-density environments and underscore its potential for real-world deployment.