Closed form for $\sum_{k=1}^\infty(\zeta(4k+1)-1)$ Wikipedia gives
$$\sum_{k=2}^\infty(\zeta(k)-1)=1,\quad\sum_{k=1}^\infty(\zeta(2k)-1)=\frac34,\quad\sum_{k=1}^\infty(\zeta(4k)-1)=\frac78-\frac\pi4\left(\frac{e^{2\pi}+1}{e^{2\pi}-1}\right)$$
from which we can easily find $\sum_{k=1}^\infty(\zeta(2k+1)-1)$ and $\sum_{k=1}^\infty(\zeta(4k+2)-1)$. From here it's natural to ask the following

Question: Is there a known closed form for $\sum_{k=1}^\infty(\zeta(4k+1)-1)$?

Related: Closed form for $\sum\limits_{k=1}^{\infty}\zeta(4k-2)-\zeta(4k)$ and Closed form for $\sum_{k=1}^{\infty} \zeta(2k)-\zeta(2k+1)$.

Progress
Note that we are done once we have a closed form for $\sum_{k=1}^\infty(\zeta(4k+1)-\zeta(4k+3))$. So I tried the same approach as in one of the questions above and this is what I got:
We have $$\zeta(4k+1)-\zeta(4k+3)=\sum_{n\geq2}^\infty\left(1-\frac1{n^2}\right)\frac1{n^{4k+1}}$$
Hence 
$$\sum_{k\geq1}\zeta(4k+1)-\zeta(4k+3)=\sum_{n\geq2}\left(1-\frac1{n^2}\right)\sum_{k\geq1}\frac1{n^{4k+1}}=\sum_{n\geq2}\left(1-\frac1{n^2}\right)\frac1{n^5}\frac1{1-\frac1{n^4}}\\=\sum_{n\geq2}\frac1{n^3+n^5}.$$
In the same way I get $$\sum_{k\geq1}\zeta(4k+1)-\zeta(4k)=-\sum_{n\geq2}\frac1{n+n^2+n^3+n^4}$$ and $$\sum_{k\geq1}\zeta(4k+1)-\zeta(4k+2)=\sum_{n\geq2}\frac1{n^2+n^3+n^4+n^5}.$$
It suffices (in fact it is equally hard) to evaluate any of these series.

By expressing the infinite sum as a double sum and then switching the order of summation, we get
$$\begin{align} \sum_{n=1}^{\infty} \left[\zeta(4n+1) - 1 \right] &= \sum_{n=1}^{\infty} \sum_{m=2}^{\infty} \frac{1}{m^{4n+1}} = \sum_{m=2}^{\infty} \frac{1}{m}\sum_{n=1}^{\infty} \left( \frac{1}{m^{4}} \right)^{n} \\ &= \sum_{m=2}^{\infty} \frac{1}{m} \frac{\frac{1}{m^{4}}}{1-\frac{1}{m^{4}}} = \sum_{m=2}^{\infty} \frac{1}{m} \frac{1}{m^{4}-1}. \end{align}$$
One can use the series representation $$\psi(2+z) = - \gamma +1 + \sum_{n=2}^{\infty} \frac{z}{n(n+z)}, \tag{1}$$ where $\psi(x)$ is the digamma function, to show
$$\psi(2+z) + \psi(2-z) + \psi(2+iz) + \psi(2-iz) = - 4 \gamma +4 -4z^{4} \sum_{n=2}^{\infty} \frac{1}{n(n^{4}-z^{4})}.$$
Letting $z=1$, we find
$$ \begin{align} \sum_{m=2}^{\infty} \frac{1}{m(m^{4}-1)} &= 1-\gamma - \frac{1}{4} \Big(\psi(3) + \psi(1) + \psi(2+i) + \psi(2-i) \Big) \\ &= 1-\gamma  - \frac{1}{4} \left(\frac{3}{2} - \gamma - \gamma + \psi(2+i)+\psi(2-i) \right) \\ &= \frac{5}{8}-\frac{\gamma}{2}  - \frac{1}{4} \Big(\psi(2-i)+\psi(2+i) \Big) \\ &\approx 0.0390670072. \end{align}$$
EDIT:
$(1)$ I find it easier to prove that series representation for $\psi(2+z)$ using the difference equation $$\psi(N+1+z) - \psi(z) = \sum_{n=0}^{N} \frac{1}{n+z} $$ as opposed to manipulating the series representation for $\psi(1+z)$.
$$ \begin{align} \sum_{n=2}^{N} \frac{z}{n(n+z)} &= \sum_{n=2}^{N} \frac{1}{n} - \sum_{n=2}^{N} \frac{1}{n+z} \\ &= \Big( H_{N}-1 \Big) - \Big(\psi(N+1+z) - \psi(z) - \frac{1}{z} - \frac{1}{1+z} \Big)\\ &= \Big( \psi(N+1) + \gamma -1 \Big) - \Big(\psi(N+1+z) - \psi(2+z) \Big) \end{align}$$
Letting $N \to \infty$ leads to the result since $\psi(x+a) = \ln(x) + \mathcal{O} \left(\frac{1}{x} \right)$ as $x \to \infty$.
