Keras Mxnet CustomMetric

Discussion
1

Hi,

I am running a Unet implementation in Keras (originally tensorflow) with an mxnet backend. It has an iou custom metric that is registered in tf using the py_func op. I tried wrapping the metric function in mx.metric.create, but I am getting:

ValueError: ('Could not interpret metric function identifier:', <mxnet.metric.CustomMetric object at 0x7f4a35350518>)

I haven’t found any information on keras mxnet supporting custom metrics yet. Can I use a CustomMetric with the keras-mxnet backend?

2

Hi @cory,

Would be great if you could share the definition for the custom metric? I’m interested to see whether it’s defined using keras.backend operators (or could be). You should then be able to reference the function directly when compiling the model:

model.compile(optimizer='abc', loss='xyz', metrics=[iou_fn])
3

Hi @thomelane,

Definitely! Here is the original:

def get_iou_vector(A, B):
    batch_size = A.shape[0]
    metric = []
    for batch in range(batch_size):
        t, p = A[batch]>0, B[batch]>0
#         if np.count_nonzero(t) == 0 and np.count_nonzero(p) > 0:
#             metric.append(0)
#             continue
#         if np.count_nonzero(t) >= 1 and np.count_nonzero(p) == 0:
#             metric.append(0)
#             continue
#         if np.count_nonzero(t) == 0 and np.count_nonzero(p) == 0:
#             metric.append(1)
#             continue
        
        intersection = np.logical_and(t, p)
        union = np.logical_or(t, p)
        iou = (np.sum(intersection > 0) + 1e-10 )/ (np.sum(union > 0) + 1e-10)
        thresholds = np.arange(0.5, 1, 0.05)
        s = []
        for thresh in thresholds:
            s.append(iou > thresh)
        metric.append(np.mean(s))

    return np.mean(metric)

def my_iou_metric(label, pred):
    return tf.py_func(get_iou_vector, [label, pred>0.5], tf.float64)

def my_iou_metric_2(label, pred):
    return tf.py_func(get_iou_vector, [label, pred >0], tf.float64) 
...
model1.compile(loss="binary_crossentropy", optimizer=c, metrics=[my_iou_metric])
...
model_checkpoint = ModelCheckpoint(save_model_name,monitor='my_iou_metric', 
                               mode = 'max', save_best_only=True, verbose=1)
reduce_lr = ReduceLROnPlateau(monitor='my_iou_metric', mode = 'max',factor=0.5, patience=5, min_lr=0.0001, verbose=1)

epochs = 50
batch_size = 32
history = model1.fit(x_train, y_train,
                    validation_data=[x_valid, y_valid], 
                    epochs=epochs,
                    batch_size=batch_size,
                    callbacks=[ model_checkpoint,reduce_lr], 
                    verbose=2)

Haha @roywei had the same advice earlier today. Here is what I have so far:

def get_iou_vector_batch_iter(t, p):
    """
    union:
    >>> np.array([[0.5, 0.1], [0.7, 0.0], [0.4, 1.5]])>0.5
    array([[False, False],
           [ True, False],
           [False,  True]])
    >>> (np.array([[0.5, 0.1], [0.7, 0.0], [0.4, 1.5]])>0.5)+
                np.array([[False, False], [False, True], [False, True]]) 
    array([[False, False],
          [ True,  True],
          [False,  True]])
    >>>
    """
    intersection = t&p #np.logical_and(t, p)
    union = t+p #np.logical_or(t, p)
    #iou = (np.sum(intersection > 0) + 1e-10 )/ (np.sum(union > 0) + 1e-10)
    iou = (K.sum(intersection > 0) + 1e-10) / (K.sum(union > 0) + 1e-10)
    #thresholds = np.arange(0.5, 1, 0.05)
    thresholds = K.arange(0.5, 1, 0.05)
    #s = []
    #for thresh in thresholds:
    #    s.append(iou > thresh)
    s = K.map_fn(lambda thresh: iou > thresh, thresholds)
    return K.mean(s) #np.mean(s))

def get_iou_vector(A, B):
    batch_size = A.shape[0]
    #metric = []
    #for batch in range(batch_size):
    #    metric.append(get_iou_vector_batch_iter(A[batch]>0, B[batch]>0))
    metric = K.map_fn(lambda batch: get_iou_vector_batch_iter(A[batch]>0, B[batch]>0), 
                      K.arange(0, batch_size))
    return K.mean(metric) #np.mean(metric)

def my_iou_metric_50(label, pred):
    return get_iou_vector(label, pred>0.5)

def my_iou_metric_0(label, pred):
    return get_iout_vector(label, pred>0)

Not sure if a few things work as intended since I haven’t started runtime debugging as I hit:

NotImplementedError: MXNet Backend: map_fn operator is not supported yet.

Do you have any suggestions around this and do you see any errors in my initial code? For example, will t&p work? I don’t see it in the comparison operators on https://mxnet.incubator.apache.org/api/python/symbol/symbol.html?highlight=infer_shape#symbol-api

Thanks again!

4

Ah, okay, so I’ve raised an issue on Github about this so the Keras-MXNet team can take a look.

5

Thanks for the feedback, updated solution in issue

6

Thanks @roywei’s solution worked!