  In this paper we propose an algorithm for polynomial-time reinforcement
learning in factored Markov decision processes (FMDPs). The factored optimistic
initial model (FOIM) algorithm, maintains an empirical model of the FMDP in a
conventional way, and always follows a greedy policy with respect to its model.
The only trick of the algorithm is that the model is initialized
optimistically. We prove that with suitable initialization (i) FOIM converges
to the fixed point of approximate value iteration (AVI); (ii) the number of
steps when the agent makes non-near-optimal decisions (with respect to the
solution of AVI) is polynomial in all relevant quantities; (iii) the per-step
costs of the algorithm are also polynomial. To our best knowledge, FOIM is the
first algorithm with these properties. This extended version contains the
rigorous proofs of the main theorem. A version of this paper appeared in
ICML'09.

Artificial Intelligence;Machine Learning