Q:

Taking derivative with respect to a vector x

$\textbf{x}$ is a n-by-1 vector, $F(\textbf{x})$ is function from $R^{n} \rightarrow R^{n\times n}$.
What is the derivative of $F(\textbf{x})\textbf{x}$ with respect to $\textbf{x}$?
$\dfrac{dF(\textbf{x})\textbf{x}}{d\textbf{x}}$ = ?

A:

You can write it in index notation as
$$ \left[\frac{d\left[F(\mathbf{x})\mathbf{x}\right]}{d\mathbf{x}}\right]_{ik} = \frac{\partial\left(F_{ij}x_j\right)}{x_k} = \frac{\partial F_{ij}}{\partial x_k} x_j + \delta_{jk} F_{ij} = \frac{\partial F_{ij}}{\partial x_k} x_j + F_{ik}$$
where I've assumed sums over matching indices and $\delta_{jk}$ is the Kronecker delta function. 
EDIT: To make clear what the first term looks like explicitly in matrix notation, take $\left[F(\mathbf{x})\right]_{ij}$ to be a matrix element of $F(\textbf{x})$ and $x_j$ to be an element of the vector $\mathbf{x}$. There is a sum over $j$ and this becomes the product of a matrix and a vector:
$$ \frac{\partial F_{ij}}{\partial x_k} x_j \longrightarrow \begin{pmatrix} \displaystyle{\frac{\partial \left[F(\mathbf{x})\right]_{00}}{\partial x_k}} & \displaystyle{\frac{\partial \left[F(\mathbf{x})\right]_{01}}{\partial x_k}} & \cdots \\ \displaystyle{\frac{\partial \left[F(\mathbf{x})\right]_{10}}{\partial x_k}} & \displaystyle{\frac{\partial \left[F(\mathbf{x})\right]_{11}}{\partial x_k}} & \cdots \\ \vdots & \vdots & \ddots\end{pmatrix}\begin{pmatrix}x_0 \\ x_1 \\ \vdots\end{pmatrix}$$
This is an $n$-dimensional vector and $x_k$, which we are taking derivatives with still has a free index, so this produces an $n\times n$ dimension result, as expected.

