There are generally a lot fewer vowels than consonants in the phoneme inventory of human languages. That means, with fewer sounds you need to make the words a lot longer if you want to have a decent-sized vocabulary.
Also, vowel pronunciation is more varied. There aren't many different ways to pronounce /t/ or /k/, but any regional dialect will change vowels. Partly because there are no fixed places of articulation: vowels are distinguished by the opening of the jaw and the position of the tongue. Both of these are almost infinitely variable compared to closing your lips and releasing them (as you'd do for a /p/ or a /b/).
And finally, why would you disregard most potential sounds that the human vocal apparatus can produce? It's just not an efficient use of your anatomy. A bit like hopping on one leg instead of walking.
So the two main arguments against a vowel-only language are size of phoneme inventory and variability of vowel pronunciation.