The new 802.11 amendment, 802.11ax, represents a significant shift in the WLAN operation, specially in the MAC layer where the access mechanism is now OFDMA. In particular, the Access Point (AP) is now responsible for scheduling the terminals’ transmissions, which avoids collisions and results in an efficient usage of the spectrum. However, a full transition to this new technology is not foreseeable for several years, and until then mixed scenarios that also include legacy stations will be predominant. In this context, where both the AP and the legacy stations use CSMA/CA to access the channel, a very challenging aspect is the coexistence between both types of stations, where naturally the AP should have priority but legacy stations should not be excluded. In this paper we present a deep reinforcement learning system that adjusts the contention window so as to maximize a certain notion of fairness. Differently to previous proposals, none of which to the best of our knowledge focused on this mixed scenario, the choice of parameters that characterize the environment is informed on existing 802.11 models. This results for instance in a stable choice of the contention window and larger throughputs. Thorough simulations corroborate the performance of the proposed method, which we make available at https://github.com/ffrommel/RLinWiFi.