  Market making (MM) has attracted significant attention in financial trading
owing to its essential function in ensuring market liquidity. With strong
capabilities in sequential decision-making, Reinforcement Learning (RL)
technology has achieved remarkable success in quantitative trading.
Nonetheless, most existing RL-based MM methods focus on optimizing single-price
level strategies which fail at frequent order cancellations and loss of queue
priority. Strategies involving multiple price levels align better with actual
trading scenarios. However, given the complexity that multi-price level
strategies involves a comprehensive trading action space, the challenge of
effectively training profitable RL agents for MM persists. Inspired by the
efficient workflow of professional human market makers, we propose Imitative
Market Maker (IMM), a novel RL framework leveraging both knowledge from
suboptimal signal-based experts and direct policy interactions to develop
multi-price level MM strategies efficiently. The framework start with
introducing effective state and action representations adept at encoding
information about multi-price level orders. Furthermore, IMM integrates a
representation learning unit capable of capturing both short- and long-term
market trends to mitigate adverse selection risk. Subsequently, IMM formulates
an expert strategy based on signals and trains the agent through the
integration of RL and imitation learning techniques, leading to efficient
learning. Extensive experimental results on four real-world market datasets
demonstrate that IMM outperforms current RL-based market making strategies in
terms of several financial criteria. The findings of the ablation study
substantiate the effectiveness of the model components.

Machine Learning;Artificial Intelligence;Trading and Market Microstructure