Providing Quality of Service (QoS) has always been an important task for Internet Service Providers. However, the proliferation of new multimedia content services has turned it a vital and challenging issue. The problem with QoS in nowadays Internet is what to measure and how to do it in order to provide real quality levels to end-users. Recent works in the field have focused on the service consumer, assessing the QoS as perceived by the end-user. This paper addresses the automatic evaluation of the QoS as Perceived by an end-user (PQoS) of a multimedia service. We present a general overview of the PQoS approach, studying the impact of different network and multimedia features on the quality as experienced by human beings. We develop an original software tool that integrates all the aspects related to the automation of the estimation process, using a broad group of PQoS methodologies. To date and to the best of our knowledge, there is no open source software implementation that completely estimates the PQoS for a VoIP and VideoIP service in a real environment. Using this software tool and real subjective tests, we perform an unbiased comparison of the different proposed techniques for video and audio services over IP.