Q:

How does focus on JButtons actually work in Java?

I found a strange anomaly in Java Swing.
The first JButton added to the UI chronologically always fires when the uses presses the space bar, assuming he hasn't clicked another button before doing that. This behavior even occurs if getRootPane().setDefaultButton(JButton) and JButton.requestFocus() are called.
When requesting focus on a JButton there seem to be at least 2 different kinds of "focus".
One of the "focusses" or highlightings is a dashed rectangle around the text on the button, while the other one is a thicker outline around the specified button.
The button with the dashed outlined text fires whenever the space bar is pressed.
The button with the thick border fires whenever the enter key is pressed.
I prepared a compilable minimal example illustrating this behaviour. There is no key mapping/binding involved at all.
import java.awt.Container;
import java.awt.Dimension;
import java.awt.Insets;
import java.awt.Toolkit;
import java.awt.event.ActionEvent;
import javax.swing.JButton;
import javax.swing.JDialog;
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.JTextField;
import javax.swing.WindowConstants;

public class ButtonFocusAnomalyExample extends JFrame {
    public ButtonFocusAnomalyExample() {
        super();
        setDefaultCloseOperation(WindowConstants.DISPOSE_ON_CLOSE);
        int frameWidth = 300;
        int frameHeight = 300;
        setSize(frameWidth, frameHeight);
        Dimension d = Toolkit.getDefaultToolkit().getScreenSize();
        int x = (d.width - getSize().width) / 2;
        int y = (d.height - getSize().height) / 2;
        setLocation(x, y);
        setTitle("Any Frame");
        setResizable(false);
        Container cp = getContentPane();
        cp.setLayout(null);
        setVisible(true);
        new DialogMinimal(this, true); // Runs the Dialog
    }

    public static void main(String[] args) {
        new ButtonFocusAnomalyExample();
    }

    static class DialogMinimal extends JDialog {
        private final JTextField output = new JTextField();

        public DialogMinimal(final JFrame owner, final boolean modal) {
            super(owner, modal);
            setDefaultCloseOperation(WindowConstants.DISPOSE_ON_CLOSE);
            int frameWidth = 252;
            int frameHeight = 126;
            setSize(frameWidth, frameHeight);
            Dimension d = Toolkit.getDefaultToolkit().getScreenSize();
            int x = (d.width - getSize().width) / 2;
            int y = (d.height - getSize().height) / 2;
            setLocation(x, y);
            setTitle("Minimal Button Focus Example");
            Container cp = getContentPane();
            cp.setLayout(null);
            JButton bYes = new JButton();
            bYes.setBounds(0, 0, 100, 33);
            bYes.setText("Yes (Space)");
            bYes.addActionListener(this::bYes_ActionPerformed);
            JPanel buttonPanel = new JPanel(null, true);
            buttonPanel.add(bYes);
            JButton bNo = new JButton();
            bNo.setBounds(108, 0, 120, 33);
            bNo.setText("No (Enter/Return)");
            getRootPane().setDefaultButton(bNo); // Set "No" as default button
            bNo.requestFocus(); // Get focus on "No" button
            bNo.addActionListener(this::bNo_ActionPerformed);
            buttonPanel.add(bNo);
            buttonPanel.setBounds(8, 8, 400, 92);
            buttonPanel.setOpaque(false);
            cp.add(buttonPanel);
            output.setBounds(8, 50, 220, 32);
            cp.add(output);
            setResizable(false);
            setVisible(true);
        }

        public void bYes_ActionPerformed(final ActionEvent evt) {
            output.setText("Yes"); // Still fires on every space bar press
        }

        public void bNo_ActionPerformed(final ActionEvent evt) {
            output.setText("No"); // Only fires on every return/enter press
        }
    }
}

This is what it looks like:

The executable code can also be found here.
My questions now are:

What are these different focusses?
How can someone change the focus that shows as a dashed outline around the text of the button so that the space bar and the enter key would fire the event of the "No" button?

A:

Regarding question 1:
There are no 2 kinds of "focus". Both methods do what their respective names say:
JButton.requestFocus() (better yet JButton.requestFocusInWindow()) requests focus on a button, while getRootPane().setDefaultButton(JButton) sets a selected button, which the LAF handles seperately.

Regarding question 2:
The modality of the Dialog is the problem.
Possible solutions would therefore be to either

Set modality to false when creating the Dialog, e.g. with new DialogMinimal(this, false); and get focus by calling bNo.requestFocusInWindow() instead of getRootPane().setDefaultButton(bNo); and/or bNo.requestFocus();, but this is no solution if the Dialog has to be modal.
Implement RequestFocusListener found in Dialog Focus as suggested by user camickr.

public DialogMinimal(final JFrame owner, final boolean modal) {
    Button bNo = new JButton();
    [...]
    // bNo.requestFocusInWindow(); // obsolete now
    getRootPane().setDefaultButton(bNo); // To fire on enter key
    bNo.addAncestorListener(new RequestFocusListener()); // To fire on space bar
    [...]
}

