Existence of solution for a system of quadratic diophantine equations / symmetric quadratic froms I am interested in solving, or even just deciding the existence of a solution, for a system of quadratic diophantine equations.
Let $p$ be a prime congruent to 1 modulo 8, so $ p =17$ is the first case. We want to solve the following equations inside the integers.
Let $\alpha_1,\alpha_2,...,\alpha_p$ and $\beta_1,\beta_2,...,\beta_p$ be unknowns. We understand the indices of the $\alpha$'s and $\beta$'s modulo $p$, so that e.g. $\alpha_{-1} = \alpha_{p-1}$. We have the linear dependence between the unknowns:
\begin{align}
& \sum_{j = 1}^p \alpha_j = 0, \\
& \sum_{j = 1}^p \beta_j = 0 
\end{align}
Moreover for all $i$ between $1$ and $p-1$ we have two quadratic equations given as:
\begin{align}
& \sum_{j=1}^p \alpha_j\beta_{j-i} + \alpha_{j-i}\beta_j = 0, \\ 
& \sum_{j=1}^p - \alpha_j\alpha_{j-i} + \beta_j\beta_{j-i} = \left\{ \begin{array}{ll} 2, & i = \pm 2 \\  0, & i \neq \pm 2 \end{array}\right.
\end{align}
It is easy to see that the equations for $\pm i$ are actually the same.
Additionally we have conditions on the parities:
\begin{align}
& \alpha_1 \equiv \alpha_{p-1} \equiv \beta_1 \equiv \beta_{p-1} \equiv 1 \mod 2, \\
& \alpha_i \equiv \beta_i \equiv 0 \mod 2, \ \ \text{if} \ \ i \neq \pm 1
\end{align}
Alternative formulation: The problem can also be formulated using symmetric quadratic forms. For this let $A$ be the $p \times p$-permutation matrix
$$\begin{pmatrix} 0 & 0 &  \cdots & 0 & 1 \\  1 & 0 &  \cdots & 0 & 0 \\ 0 & 1 &  \cdots & 0 & 0 \\ \vdots & \vdots &  \ddots & \vdots & \vdots \\ 0 & 0 &  \cdots & 1 & 0  \end{pmatrix} $$
Note that $A$ has order $p$. Let $B_i = A^i + (A^i)^T$ for each $1 \leq i \leq \frac{p-1}{2}$. Writing in block form, in particular $0$ for a $0$-matrix, define for $1 \leq i \leq \frac{p-1}{2}$ the quadratic forms
$$Q_i: \mathbb{Z}^{2p} \rightarrow \mathbb{Z}, \ \ x \mapsto x^T \begin{pmatrix} 0 & B_i \\ B_i & 0 \end{pmatrix}x $$
and
$$R_i: \mathbb{Z}^{2p} \rightarrow \mathbb{Z}, \ \ x \mapsto x^T \begin{pmatrix} -B_i & 0 \\ 0 & B_i \end{pmatrix}x $$
Then we can formulate the equations as
$$Q_i(x) = 0 \ \ \text{and} \ \ R_i(x) = 4\delta_{i,2} $$
for all $1 \leq i \leq \frac{p-1}{2}$ where $\delta_{i,j}$ is the Kronecker delta.
If this is of any help, we can completely solve the equations modulo $4$. This gives that
\begin{align}
& \alpha_1\beta_{p-1} + \alpha_{p-1}\beta_1 \equiv 0 \mod 4, \\
& \alpha_i + \beta_i + \alpha_{-i} + \beta_{-i}  \equiv 0 \mod 4  \ \ \text{if} \ \ i \neq 0, \pm 1 \\
& \alpha_0 + \beta_0 \equiv 2 \mod 4
\end{align}
By the origin of the problem from a question on group rings, we also know that solutions exist when p is congruent to 3 modulo 4, but we have no clue when p is congruent to 1 modulo 8. We have tried some computer experiments, but found no solution and the system seems too big for a complete solution by the programs we tried.

UPDATE. Using factorization $-2(x+1)^2x^{p-3}$ over the corresponding number field, I established that there are no solutions for $p=17$. Furthermore, I computationally verified that for primes $p<30$ we have solutions for all $p\equiv 3\pmod{4}$ and do not have any for $p\equiv 1\pmod{4}$.

This is just an extended comment, giving reformulation of the problem and reducing it to just $p-1$ unknowns and $p-1$ quadratic equations over the Gaussian integers.
Consider the generating polynomials:
\begin{split}
A(x) &:= \sum_{i=0}^{p-1} \alpha_i x^i, \\
B(x) &:= \sum_{i=0}^{p-1} \beta_i x^i.
\end{split}
The linear equations $\sum_j \alpha_j = \sum_j \beta_j = 0$ are equivalent to $A(1)=B(1)=0$, i.e., both $A(x)=(x-1)\bar A(x)$ and $B(x)=(x-1)\bar B(x)$ are multiples of $x-1$.
Viewing indices modulo $p$ is equivalent to viewing the polynomials modulo $x^p - 1 = (x-1)\Phi_p(x)$, where $\Phi_p(x) := 1 + x + \dots + x^{p-1}$ is $p$-th cyclotomic polynomial.
For reciprocal polynomials (of fixed degree $p-1$) we have $A^\star(x):=x^{p-1}A(x^{-1})\equiv x^{p-1}A(x^{p-1})\pmod{x^p-1}$ and $B^\star(x):=x^{p-1}B(x^{-1})\equiv x^{p-1}B(x^{p-1})\pmod{x^p-1}$. Then the quadratic equations (under the condition $A(1)=B(1)=0$) translate into
$$\begin{cases}
A(x)B^\star(x) + A^\star(x)B(x) \equiv 0 \pmod{x^p-1},\\
-A(x)A^\star(x) + B(x)B^\star(x) \equiv -4x^{p-1} + 2x + 2x^{p-3} \equiv 2(x^2-1)^2x^{p-3} \pmod{x^p-1}
\end{cases}
$$
Dividing both congruences by $(x-1)x(\frac1x-1)=-(x-1)^2$, we get
$$\begin{cases}
\bar A(x)\bar B^\star(x) + \bar A^\star(x)\bar B(x) \equiv 0 \pmod{\Phi_p(x)},\\
-\bar A(x)\bar A^\star(x) + \bar B(x)\bar B^\star(x) \equiv -2(x+1)^2x^{p-3} \pmod{\Phi_p(x)}.
\end{cases}
$$
In terms of polynomials over Gaussian integers, we have
$$F(x)F^\star(x) \equiv -2(x+1)^2x^{p-3}\pmod{\Phi_p(x)},$$
where
$$F(x) := \bar B(x) + I\cdot \bar A(x)$$
is a polynomial of degree $p-2$ over the Gaussian integers.
The last congruence can be viewed as a system of $p-1$ quadratic equations on the coefficients of $F(x)$ as unknowns.
Alternatively, it can also be viewed as the identity of palindromic polynomials:
$$F(x)F^\star(x) + 2(x+1)^2x^{p-3} = G(x)\cdot \Phi_p(x),$$
where the left-hand side, $G(x)$, and $\Phi_p(x)$ are palindromic polynomials of degree $2p-4$, $p-3$, and $p-1$, respectively.
