I have two classes: first one is the main QMainWindow class, and the second one is my custom class. For example, I want to make a connection in the constructor of my custom class where when I press a TestButton (which is a part of the ui of main class), it calls a function from my custom class.
Here are code:
Program.h:
class Custom;

class Program : public QMainWindow
{
    Q_OBJECT

    friend class Custom;

public:
    Program(QWidget *parent = nullptr);
    ~Program();

private:
    Ui::ProgramClass ui;
}

Program.cpp:
#include "Program.h"
#include "Custom.h"

Program::Program(QWidget *parent)
    : QMainWindow(parent)
{
    ui.setupUi

    Custom custom = Custom(this);
}

Custom.h:
#include "Program.h"

class Custom : public QObject {
    Q_OBJECT
public:
    Custom(Program* program);
    ~Custom();

public slots:
    void foo();

private:
    Program* m_program;
}

and finally Custom.cpp:
#include "Custom.h"
#include "Program.h"

Custom::Custom(Program* program) {
    m_program = program;

    /* Here's the main problem */
    QObject::connect(m_program->ui.TestButton, &QPushButton::clicked, m_program, &Custom::foo);
}

/* Here just changing text of the button and set flat true or false every time button is clicked */
void Custom::foo() {
    QPushButton* button = m_program->ui.TestButton;

    button->setFlat(!button->isFlat());
    button->setText(button->isFlat() ?
        "If you see the text changing when clicking the button, it means test is working correctly" :
        "Text changed");
}

The main part is in the Custom constructor, where I typed connect function.
Error: cannot convert argument 3 from 'Program *' to 'const Custom *.
So, a pointer to receiver and to function foo must be the same classes.
So I tried this:
QObject::connect(m_program->ui.TestButton, &QPushButton::clicked, this, &Custom::foo);
No error, but there is actually no connection to the main program and when I click the button - nothing changing.
The only working variants are:

make foo function as a method of Program class, BUT I don't want to make Program class have a lot of functions which are actually should be methods of separated classes. And if I want to modify some other fields of these separated classes, this variant won't work;
type lambda-function inside QObject::connection, BUT I have some big functions, and I need to call some of them as a slot in QObject::connect oftently.

So, how can I make a proper connection to the whole program but leave foo function as the method of Custom class?

Qt QObject::connect receiver and slot of different classes
The line  Custom custom = Custom(this); creates a local object which "dies immediately" after exit from Program constructor, that's not what you want to do. This is what you have to do for an instance of Custom to persist:
 Custom *custom = new Custom(this);

You can even make pointer named custom a member variable if you want access it later. The constructor of Custom must be:
 Custom::Custom(Program* program) : QObject(program) 
 {
     QObject::connect( m_program->ui.TestButton, &QPushButton::clicked,
                       this, &Custom::foo );
 }

This constructor passes pointer to constructor of QObject, which registers Custom as a "child" of provided object, a Program in our case. In Qt terms Program will be responsible for Custom instance's destruction.
Is what you meant to do? To connect a button to an instance of Custom? Frankly, using m_program->ui.TestButton here invades Programs personal space and relies on implementation, but it's an offtopic here.
But let's make a step aside and take a look why actually what you did, even if that was utterly wrong, didn't work?
Let's put aside functions and slots. What you'd do if had to do  this with "normal" classes and to call a method foo() of class B using a pointer of distinct class A?
Right, class B should be derived from A and foo() should be a virtual method first declared in A. THis allows a kind of type erasure where  a pointer  or reference to B can be passed as pointer to A. Functions can be passed by pointer too, but not if they are members of a class. For that a special kind of pointer exist.
#include <iostream>

class A {
public:
   virtual void foo() = 0;
};

class B : public A {
public:
   virtual void foo() { std::cout << "Hello from foo()!\n"; }
};

// A registering\calling class
class C {
public:

   void connect(A* p) {  
         cptr = p;
         f_ptr = &A::foo;
   }

   void call() {  (cptr->*f_ptr)(); }
private:
   A    *cptr;
   void (A::*f_ptr)();
};

int main() {
    B  b;
    C  c;
    c.connect(&b);
    c.call();
}

Now, you must understand what C::connect does there: it saves value of pointer to the object A* p and a pointer to the member. The expression &A::foo, a pointer to a member of A, is legal and correct for overridden &B::foo if we will use it with a pointer to B.
The pointer-to-member could be made a parameter of C::connect like  in Qt, but to make it work with any member function we need to create a template and another level of type erasure to save those values. I left it out for brevity.
It's almost same what happens with Qt signal\slot system, at least if you use direct connection and new connect syntax. You need a class instance in order to call its member, that's why connect got such syntax. Even if you had succeed in performing connection, you would have invoked an Undefined Behavior by clicking connected button. Thankfully, Qt handles it gracefully by disconnecting destroyed objects, therefore nothing actually happens.
That's why all classes that use signal\slot have to be descendants of QObject. Signals and slots are just functions. The difference between them is that meta object compiler generates an implementation for signals.
For type erasure to work, you can do either of those:

have to pass a pointer of type QObject* to an instance of Custom and cast it to class Custom in order for QObject::connect to work. The cast is unnecessary if signal or virtual slot is declared in QObject.
Or you have to pass a pointer to some base class which already got the slot void foo() declared.

You can declare a slot as virtual and you don't need to do so for signals.
public slots:
    virtual void foo();  // can be pure in abstract class

It appears to me that to have a QMainWindow as a base class is a bad idea, you'd needs some third class. In simplest cases it creates too verbose code and that's why another overload was introduced, which allows a lambda or functor object.
