We consider a routing overlay in which the delay of a path can be obtained at some fixed cost by sending probe packets, and investigate the joint minimization of the probing cost and the routing delay. Assuming that link delays are modelled by Markov chains, this problem can be cast as a Markov Decision Process (MDP). Unfortunately, computing the exact solution of this MDP is prohibitively expensive due to the well-known "curse of dimensionality". In this work we propose two scalable approaches that are fast enough to provide efficient solutions on practical time scales. We analyze the complexity of both approaches, and evaluate their accuracy in small synthetic scenarios for which the optimal monitoring policy can be computed. Finally, the robustness and the scalability of the proposed solutions are analyzed using real delay data collected over the Internet.