In addition to this example: callback = keras.callbacks.EarlyStopping(monitor='val_loss')

Allow monitoring of multiple metrics, as in this example: callback = keras.callbacks.EarlyStopping(monitor=['val_loss', 'val_accuracy', 'val_f1measure'])

This way, training should not stop while any of these metrics get better values, not just one of them.

Comment From: rlcauvin

One factor to consider when using multiple metrics for early stopping: how do you determine which epoch is "best" and should be used for restoring best weights (when restore_best_weights is True) after early stopping?

Comment From: fabriciorsf

The best epoch will be the one with the best val_loss value, but patience should not consider just one metric, but rather a list of metrics passed in the monitor parameter. Another possibility would be to consider the best epoch by looking at the value of the first metric in the list passed to the monitor parameter, in case of a tie, look at the second, and so on. Thank you for your quick response.

Comment From: rlcauvin

Another consideration is that we want to minimize some metrics (e.g. val_loss) but maximize other metrics (e.g. val_accuracy).

It might be instructive to look at this implementation of a composite of early stopping callbacks. The iterable_condition enables the caller to determine whether all, any, or some other combination of early stopping conditions must hold. It assumes the last early stopping callback is the "conductor", which means it determines which epoch is "best".

class CompoundEarlyStopping(keras.callbacks.Callback):
  def __init__(
    self,
    callbacks: Iterable[keras.callbacks.Callback],
    iterable_condition: Callable[Iterable, bool] = all):

    super().__init__()

    self.callbacks = callbacks
    self.stopped_epoch = 0
    self.iterable_condition = iterable_condition

  def on_train_begin(
    self,
    logs: Dict = None):

    for callback in self.callbacks:
      callback.on_train_begin(logs)

  def on_train_end(
    self,
    logs: Dict = None):

    if self.model.stop_training:
      conductor = self.callbacks[-1]
      conductor.stopped_epoch = self.stopped_epoch
      conductor.on_train_end(logs)

  def on_epoch_begin(
    self,
    epoch: int,
    logs: Dict = None):

    for callback in self.callbacks:
      callback.on_epoch_begin(epoch, logs)

  def on_epoch_end(
    self,
    epoch: int,
    logs: Dict = None):

    for callback in self.callbacks:
      callback.on_epoch_end(epoch, logs)

    if self.iterable_condition([callback.stopped_epoch >= max(1, epoch) for callback in self.callbacks]):
      self.stopped_epoch = epoch
      self.model.stop_training = True
    else:
      self.model.stop_training = False

  def set_model(
    self,
    model: keras.Model):

    super().set_model(model)

    for callback in self.callbacks:
      callback.set_model(model)

Here is an example of how I've used the CompoundEarlyStopping class:

patience = 3

early_stopping_loss = keras.callbacks.EarlyStopping(monitor="val_loss", patience=patience, mode="min", verbose=1)

early_stopping_accuracy = keras.callbacks.EarlyStopping(
  monitor="val_accuracy",
  patience=patience,
  mode="max",
  verbose=1)

early_stopping_auc = keras.callbacks.EarlyStopping(
  monitor="val_auc",
  patience=patience,
  mode="max",
  restore_best_weights=True,
  verbose=1)

early_stopping = CompoundEarlyStopping(
  callbacks=[early_stopping_loss, early_stopping_accuracy, early_stopping_auc],
  iterable_condition=all)

Comment From: SamanehSaadat

Hi @fabriciorsf

As @rlcauvin mentioned, you can achieve this by creating a custom callback. Closing this issue. Please feel free to re-open if that's not what you're looking for.

Comment From: fabriciorsf

I tried this solution, but I had some problems:

  1. Intermittently, at the end of the first epoch the fit method freezes after calculating val_metrics;
  2. When the previous problem does not occur, this error occurs:
...
Epoch 5/100
23/23 ━━━━━━━━━━━━━━━━━━━━ 0s 98ms/step - loss: 0.3884 - sumsqeuc_dist: 0.3885
Epoch 5: saving model to ./saved_models/my_model_epoch_005.keras
23/23 ━━━━━━━━━━━━━━━━━━━━ 5s 185ms/step - loss: 0.3883 - sumsqeuc_dist: 0.3884 - val_loss: 0.9949 - val_sumsqeuc_dist: 0.3933
Traceback (most recent call last):
Traceback (most recent call last):
...  
File "myscript.py", line 293, in train_autonem
    history = self.my_model.fit(self.train_inputs,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "....../python3.12/site-packages/keras/src/utils/traceback_utils.py", line 122, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "..../mypackage/utils/keras_extensions.py", line 268, in on_train_end
    conductor = self.callbacks[-1]
                ~~~~~~~~~~~~~~^^^^
TypeError: 'dict_values' object is not subscriptable

The class CompoundEarlyStopping is in file keras_extensions.py:

class CompoundEarlyStopping(keras.callbacks.Callback):
    def __init__(
        self, callbacks: Iterable[Callback],
        iterable_condition: Callable[[], bool] = all):

        super().__init__()

        self.callbacks = callbacks
        self.stopped_epoch = 0
        self.iterable_condition = iterable_condition

    def on_train_begin(self, logs: dict = None):
        for callback in self.callbacks:
            callback.on_train_begin(logs)

    def on_train_end(self, logs: dict = None):
        if self.model.stop_training:
            conductor = self.callbacks[-1] ## intermittent ERROR here
            conductor.stopped_epoch = self.stopped_epoch
            conductor.on_train_end(logs)

    def on_epoch_begin(self, epoch: int, logs: dict = None):
        for callback in self.callbacks:
            callback.on_epoch_begin(epoch, logs)

    def on_epoch_end(self, epoch: int, logs: dict = None):
        for callback in self.callbacks:
            callback.on_epoch_end(epoch, logs)
        if self.iterable_condition([callback.stopped_epoch >= max(1, epoch) \
                                    for callback in self.callbacks]):
            self.stopped_epoch = epoch
            self.model.stop_training = True
        else:
            self.model.stop_training = False

    def set_model(self, model: Model):
        super().set_model(model)
        for callback in self.callbacks:
            callback.set_model(model)

Note: There is a error with Callable[Iterable, bool], so I replaced it with Callable[[], bool]

I instantiate the callbacks like this:

dict_early_stopping = {val_metric_name: EarlyStopping(monitor=val_metric_name, patience=5, 
                                                      start_from_epoch=1, verbose=self.verbose) \
                       for val_metric_name in val_metrics_name}

early_stopping = CompoundEarlyStopping(
    callbacks=dict_early_stopping.values(),
    iterable_condition=all)

As a baseline, doing early_stopping = dict_early_stopping[val_metrics_name[0]] works fine.