The prediction of crystal structures is one of the most essential challenges in designing novel functional materials. A data-driven prediction technique that uses the database of known crystal structures and substitutes ions among materials of known crystal structures to concoct new crystal structures has been proposed. This technique has been applied to generate crystal-structure candidates for the purpose of first-principles-calculation-based high-throughput computational screening. However, this technique has a functional limitation that the ion substitution tendencies are available only for typical ions such that their associated crystal structures appear in well-known materials. To overcome such a limitation, this work introduces an idea of collaborative filtering to the calculation of the ionic substitution tendencies. Based on this idea, we develop symmetric matrix factorization (SMF) method to model underlying substitution conditions. In addition, we present a symmetric matrix co-factorization (SMCF) method to incorporate additional knowledge pertaining to chemical properties in estimating the substitution tendencies among ions with extremely small amount of previous knowledge in the database. The performance of the prediction is investigated along with existing techniques through in silico experiments using real crystal-structure database. The numerical results show that the proposed SMF- and SMCF-based prediction outperform existing techniques in terms of the prediction accuracy.