It's more that all the interesting things that come from prime numbers don't really work if you include 1 as a prime. So every time you talked about primes you would have to say "something something prime number, other than 1, yadda yadda".

For example: The Fundamental Theorem of Arithmetic states that every natural number can be expressed as a finite product of primes, and that prime factorization is unique.

So

12 = 2^2 x 3

is the only way to express 12 as a product of primes. But if we include 1 as a prime, then

12 = 1 x 2^2 x 3
    = 1^2 x 2^2 x 3
    = 1^3 x 2^2 x 3

etc. There are infinite prime factorization of every natural number, and the most interesting part of the Fundamental Theorem of Arithmetic no longer holds.