Matrix to Matrix sequence architecture My training data is the set of X{m x n} matricies, which have corresponding Y{m x n} output.It is like unknown function: Y = F(X) and I need to predict Y values based on X.
Cells of matrices can only have values (0,1,2).
Currently the best results for me ( acc: 0.817 , test_acc: 0.801) was provided by next snippet: ( format is 9x9 matrix -> 81 vector, train set is 100,000 samples)
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv1D(81, 9, padding="same", activation="relu"))
model.add(tf.keras.layers.Conv1D(162, 9, padding="same", activation="relu"))
model.add(tf.keras.layers.Conv1D(9, 9, padding="same", activation="relu"))
model.add(tf.keras.layers.Activation("softmax"))
model.compile(loss='mean_squared_error', optimizer='adam',metrics=['accuracy'])
model.fit(training_data, target_data, epochs=5, verbose=1, validation_data=(test_training_data,test_target_data),shuffle=True)

However after 10 epoch, this model starts overfitting.
Moreover with matrix size (256x9) accuracy is very low (~0.38) and degrades with every epoch after 5.In this case model is seriously underfitting (2/3).
I had also tried using fully-connected solution.But results are pretty similar.
model.add(tf.keras.layers.Dense(324, input_dim=81, activation='relu'))
model.add(tf.keras.layers.Dense(162, activation='relu'))
model.add(tf.keras.layers.Dense(81, activation='relu'))

These things are also provide no result.
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Dropout(rate=.1))

I was looking in such things as U-net and GAN, however I really don't think I need such things as batch size, filters, odd dimensions and so on.
Maybe there are any AutoML solutions exist for this case?

I was able to drastically improve accuracy: 1 (~9.9997) for both sets by just swapping to Conv2D with channel size 1(empty channel like grayscale image)
input_shape = (None, size_y, size_x, 1)
model.add(tf.keras.layers.Conv2D(8, kernel,padding="same", activation="relu", strides = 1, input_shape=input_shape[1:]))
model.add(tf.keras.layers.Conv2D(16 , kernel, padding="same",  strides = stride, activation="relu"))

You will need to  reshape [m,n] matrix to [batch_size, m, n, channel(empty)] via this code:
training_data = tf.expand_dims(training_data, axis=-1)

