This work investigates the use of deep learning techniques to improve the performance of web application firewalls (WAFs), systems that are used to detect and prevent attacks to web applications. Typically, a WAF inspects the HTTP requests that are exchanged between client and server to spot attacks and block potential threats. We model the problem as a one-class supervised case and build a feature extractor using deep learning techniques. We treat the HTTP requests as text and train a deep language model with a transformer encoder architecture which is a self-attention based neural network. The use of pre-trained language models has yielded significant improvements on a diverse set of NLP tasks because they are capable of doing transfer learning. We use the pre-trained model as a feature extractor to map a http request into a feature vector. These vectors are then used to train a one-class classifier. We also use a performance metric to automatically define an operational point for the one-class model. The experimental results show that the proposed approach outperforms the ones of the classic rule-based MOD- SECURITY configured with a vanilla owasp crs and does not require the participation of a security expert to define the features.