We propose an optimization and learning technique for controlling energy storage in power systems with renewables. A reinforcement learning (RL) approach is employed to bypass the need for an accurate stochastic dynamic model for wind and solar power; at the same time, the presence of the grid is explicitly accounted for through the “DC” approximation to the Optimal Power Flow (OPF) to impose line constraints. The key idea that allows the inclusion of such instantaneous constraints within the RL framework is to take as control actions the storage operational prices, which may be suitably discretized. A policy to select these actions as a function of the state is parameterized by a neural network model and trained based on traces of demand and renewables. We call this combined strategy RL-OPF. We test it on a trial network with real data records for demand and renewables, showing convergence to a control policy that induces arbitrage of energy across space and time.