Proof for recursive function:

    a(n) = n^2 - 2*n if n is even, and n^2 - n if n is odd.

    Assume a(n) = a(n-1) + 2*a(n-2) - 2*a(n-3) - a(n-4) + a(n-5).

    If a(n) is even:
    # Plug in even function for a(n-2) and a(n-4), and  plug in odd function for a(n-1), a(n-3), and a(n-5).
    a(n) = (n-1)^2 - n + 1 + 2*((n-2)^2 - 2*(n-2)) - 2*((n-3)^2 - n + 3) - (n-4)^2 + 2*(n-4) + (n-5)^2 - n + 5
    # Expand.
    a(n) = n^2 - 2*n + 1 - n + 1 + 2*n^2 - 8*n + 8 - 4*n + 8 - 2*n^2 + 12*n - 18 + 2*n - 6 - n^2 + 8*n - 16 + 2*n - 8 + n^2 - 10*n + 25 - n + 5
    # Group like terms.
    a(n) = (n^2 + 2*n^2 - 2*n^2 - n^2 + n^2) + (-2*n - n - 8*n - 4*n + 12*n + 2*n + 8*n + 2*n - 10*n - n) + (1 + 1 + 8 + 8 - 18 - 6 - 16 - 8 + 25 + 5)
    # Simplify.
    a(n) = n^2 - 2*n
    # Result is equal to even function.

    If a(n) is odd:
    # Plug in even function for a(n-1), a(n-3), and a(n-5), and  plug in odd function for a(n-2) and a(n-4).
    a(n) = (n-1)^2 - 2*(n-1) + 2*((n-2)^2 - n + 2) - 2*((n-3)^2 - 2*(n-3)) - (n-4)^2 + n - 4 + (n-5)^2 - 2*(n-5)
    # Expand.
    a(n) = n^2 - 2*n + 1 - 2*n + 2 + 2*n^2 - 8*n + 8 - 2*n + 4 - 2*n^2 + 12*n - 18 + 4*n - 12 - n^2 + 8*n - 16 + n - 4 + n^2 - 10*n + 25 - 2*n + 10
    # Group like terms.
    a(n) = (n^2 + 2*n^2 - 2*n^2 - n^2 + n^2) + (-2*n - 2*n - 8*n - 2*n + 12*n + 4*n + 8*n + n - 10*n - 2*n) + (1 + 2 + 8 + 4 - 18 - 12 - 16 - 4 + 25 + 10)
    # Simplify.
    a(n) = n^2 - n
    # Result is equal to odd function.

    The recursive definition holds up for both cases, so a(n) = a(n-1) + 2*a(n-2) - 2*a(n-3) - a(n-4) + a(n-5).

Proof for generating function:

    Using the recursive function, we can determine that a(n) - a(n-1) - 2*a(n-2) + 2*a(n-3) + a(n-4) - a(n-5) = 0.
    We can use this to figure out a generating function by adding together multiples of the generating function.
    Assume A(x) is the generating function for a(n).

                                        A(x) = 6*x^3 + 8*x^4 + 20*x^5 + 24*x^6 + 42*x^7 + 48*x^8 + 72*x^9 + 80*x^10 + 110*x^11 + 120*x^12 + 156*x^13 + O(x^14)
                                     -x*A(x) =       - 6*x^4 -  8*x^5 - 20*x^6 - 24*x^7 - 42*x^8 - 48*x^9 - 72*x^10 -  80*x^11 - 110*x^12 - 120*x^13 + O(x^14)
                                 -2*x^2*A(x) =               - 12*x^5 - 16*x^6 - 40*x^7 - 48*x^8 - 84*x^9 - 96*x^10 - 144*x^11 - 160*x^12 - 220*x^13 + O(x^14)
                                  2*x^3*A(x) =                          12*x^6 + 16*x^7 + 40*x^8 + 48*x^9 + 84*x^10 +  96*x^11 + 144*x^12 + 160*x^13 + O(x^14)
                                    x^4*A(x) =                                    6*x^7 +  8*x^8 + 20*x^9 + 24*x^10 +  42*x^11 +  48*x^12 +  72*x^13 + O(x^14)
                                   -x^5*A(x) =                                          -  6*x^8 -  8*x^9 - 20*x^10 -  24*x^11 -  42*x^12 -  48*x^13 + O(x^14)
	----------------------------------------------------------------------------------------------------------------------------------------------------------
    (1 - x - 2*x^2 + 2*x^3 + x^4 - x^5)*A(x) = 6*x^3 + 2*x^4 +  0*x^5 +  0*x^6 +  0*x^7 +  0*x^8 +  0*x^9 +  0*x^10 +   0*x^11 +   0*x^12 +   0*x^13 + O(x^14)
                                             # Because a(n) - a(n-1) - 2*a(n-2) + 2*a(n-3) + a(n-4) - a(n-5) = 0, these terms will always cancel.
                                             = 6*x^3 + 2*x^4
                                             # Divide both sides by (1 - x - 2*x^2 + 2*x^3 + x^4 - x^5)
                                        A(x) = x^3*(6 + 2*x)/(1 - x - 2*x^2 + 2*x^3 + x^4 - x^5)
                                             # Factor.
                                             = x^3*(6 + 2*x)/(-(x + 1)^2*(x - 1)^3)
                                             # Bring the negative to the top.
                                             = x^3*(-6 - 2*x)/((x + 1)^2*(x - 1)^3)

    This shows that the generating function A(x) is equal to x^3*(-6 - 2*x)/((x + 1)^2*(x - 1)^3).