1967 HSC 4 unit Mathematics Question 2 Screenshot from the examination paper
[...asking about partial fraction decomposition of $$\frac{1 - abx^2}{(1-ax)(1-bx)} $$ and related formulas...]
This question is taken from the New South Wales Higher School Certificate 4 unit (highest level possible) paper of 1967. I wasn't sure how to approach this problem. In the end, I decided to let x = 1 and this gave me u$_r$ = -1/n and R$_n$(x) = (1 - ab)x when I compared terms on left and right but I'm not sure if this is the correct approach. Any help appreciated on this one.

Let us do things formally without worrying about convergent issues. As mentioned by Jack Lam,  the partial fraction decomposition of the function is given by
\begin{align}
\frac{1-abx^2}{(1-ax)(1-b)} = \frac{1}{1-bx}+\frac{1}{1-ax}-1.
\end{align}
Formally, we will us the geometric series representation for the two fractions on the right-hand side, i.e. we have
\begin{align}
\frac{1}{1-bx} + \frac{1}{1-ax} =&\ \sum^\infty_{r=0} b^rx^r + \sum^\infty_{r=0}a^rx^r = \sum^\infty_{r=0}(b^r+a^r)x^r \\
=&\ \sum^n_{r=0}(b^r+a^r)x^r + \sum^\infty_{r=n+1} b^rx^r+\sum^\infty_{r=n+1}a^rx^r\\
=&\ \sum^n_{r=0}(b^r+a^r)x^r + b^{n+1}x^{n+1}\sum^\infty_{r=0}b^rx^r+ a^{n+1}x^{n+1}\sum^\infty_{r=0}a^rx^r\\
=&\ \sum^n_{r=0}(b^r+a^r)x^r +  \frac{b^{n+1}x^{n+1}}{1-bx}+ \frac{a^{n+1}x^{n+1}}{1-ax}.
\end{align}
Now, going back to the original function, we have
\begin{align}
\frac{1-abx^2}{(1-ax)(1-bx)} =&\  \sum^n_{r=0}(b^r+a^r)x^r +  \frac{b^{n+1}x^{n+1}}{1-bx}+ \frac{a^{n+1}x^{n+1}}{1-ax}- 1\\
=&\ \sum^n_{r=1}(b^r+a^r)x^r + \frac{b^{n+1}x^{n+1}}{1-bx}+ \frac{a^{n+1}x^{n+1}}{1-ax}+1.
\end{align}
Multiply both sides of the above equation by $(1-ax)(1-bx)$ yields
\begin{align}
1-abx^2 =&\ (1-ax)(1-bx)\left[\sum^n_{r=1}(b^r+a^r)x^r+1 \right] + [(1-ax)b^{n+1} +(1-bx)a^{n+1}]\ x^{n+1}\\
=&\ (1-ax)(1-bx)\sum^n_{r=0}u_rx^r + x^{n+1} R_n(x)
\end{align}
where 
\begin{align}
R_n(x) = (1-ax)b^{n+1} +(1-bx)a^{n+1}
\end{align}
and
\begin{align}
u_r =
\begin{cases}
1 & \text{ if } r =0,\\
a^r+b^r & \text{ if } r\geq 1
\end{cases}.
\end{align}
