Form API States required don't seem to work on a custom form with ajax

I can't get form states required to work on an ajax submitted form.

Here is some example code. It is taken from the examples module and slightly modified to illustrate the problem.

  /**
   * {@inheritdoc}
   */
  public function buildForm(array $form, FormStateInterface $form_state) {
    // This container wil be replaced by AJAX.
    $form['container'] = [
      '#type' => 'container',
      '#attributes' => ['id' => 'box-container'],
    ];
    // The box contains some markup that we can change on a submit request.
    $form['container']['box'] = [
      '#type' => 'markup',
      '#markup' => '<h1>Initial markup for box</h1>',
    ];

    $form['changethis'] = [
      '#title' => $this->t("Choose something and explain why"),
      '#type' => 'select',
      '#options' => [
        'one' => 'one',
        'two' => 'two',
        'three' => 'three',
      ],
    ];

    $form['title'] = [
      '#type' => 'textfield',
      '#title' => $this->t('Title'),
      '#states' => [
        'required' => [
          ':input[name="changethis"]' => ['value' => 'two'],
        ],
        'visible' => [
          ':input[name="changethis"]' => ['value' => 'two'],
        ],
      ],
    ];

    $form['submit'] = [
      '#type' => 'submit',
      // The AJAX handler will call our callback, and will replace whatever page
      // element has id box-container.
      '#ajax' => [
        'callback' => '::promptCallback',
        'wrapper' => 'box-container',
      ],
      '#value' => $this->t('Submit'),
    ];

    return $form;
  }

The title field shows up, when "two" is selected and is marked required. But when the form is submitted with the title field empty, neither HTML5 validation or form validation is complaining about the field being empty. When the "#ajax" part from the submit button is removed, the required state works fine.