Logloss in the recent mxnet Rpackage

Discussion
1

I used to work with an old version of mxnet for R and I used to define logloss as custom measure:

mx.metric.binarylogloss <- mx.metric.custom( "binarylogloss", function(label, pred) {
  eps = 1e-15
  p = pmin(pmax(t(pred[2,]) , eps), 1 - eps)
  res = -(sum(label * log(p) + (1 - label) * log(1 - p))) / length(label)
  return(res)
})

This does not work in the new version anymore. I get the error:

Error in pred[2, ] : object of type ‘externalptr’ is not subsettable

How can I either change the above function or directly get logloss in the R package for mxnet? Thank you!

2

@hetong007 do you have an idea?

3

Is there a reason that you only use the second column of pred ?

Besides, please use mx.nd.clip instead of pmin / pmax as in https://github.com/apache/incubator-mxnet/blob/master/R-package/R/metric.R#L98

4

@hetong007: thank you for pointing me to the measure that is already implemented in the recent version of the package. My code was from some time back when I think that a logloss function was missing for mxnet. I will simply use the one that you pointed me to. The reason why the code did not work is most probably just because the internal data structure somehow changed from the ooooold mxnet package to the present one. Back then I could take the second col and proceed as you have seen.
I will check this evening and then close the post.
Thank you and @ThomasDelteil thank you too!

5

@hetong007
I found logloss but I am afraid it does not work the following code just crashes on my machine:

library(mxnet)    
deep.model = mx.mlp(as.matrix(iris[,1:4]), ifelse(iris[,5]=="setosa",1,0),
                    eval.data          = list("data" = as.matrix(iris[,1:4]), "label" = ifelse(iris[,5]=="setosa",1,0)),
                    hidden_node        = c(20,20,20), 
                    out_node           = 2, 
                    out_activation     = "softmax",
                    num.round = 20, array.batch.size = 256, learning.rate = 0.01, momentum = 0.9, dropout = 0.8,
                    #eval.metric = mx.metric.binarylogloss, 
                    eval.metric = mx.metric.logloss, #mx.metric.logloss, mx.metric.accuracy
                    array.layout = "rowmajor",
                    activation = "relu", 
                    ctx = mx.ctx.default(),
                    initializer = mx.init.uniform(0.5),
                    epoch.end.callback = mx.callback.log.train.metric(period = 5)
)

and it does not crash using mx.metric.accuracy.

could you give me a hint? thank you!
My sessionInfo is the following (I try to make it run on R 3.5) and downloaded via
install.packages("https://github.com/jeremiedb/mxnet_winbin/raw/master/mxnet.zip", repos = NULL)

R version 3.5.0 (2018-04-23)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] mxnet_1.2.0

loaded via a namespace (and not attached):
[1] Rcpp_0.12.16 pillar_1.2.1 compiler_3.5.0 RColorBrewer_1.1-2 influenceR_0.1.0 plyr_1.8.4
[7] bindr_0.1.1 viridis_0.5.1 tools_3.5.0 digest_0.6.15 drat_0.1.4 jsonlite_1.5
[13] viridisLite_0.3.0 gtable_0.2.0 memoise_1.1.0 tibble_1.4.2 rgexf_0.15.3 pkgconfig_2.0.1
[19] rlang_0.2.0 igraph_1.2.1 rstudioapi_0.7 yaml_2.1.18 bindrcpp_0.2.2 gridExtra_2.3
[25] downloader_0.4 withr_2.1.2 DiagrammeR_1.0.0 dplyr_0.7.4 stringr_1.3.0 htmlwidgets_1.2
[31] devtools_1.13.5 hms_0.4.2 grid_3.5.0 glue_1.2.0 R6_2.2.2 Rook_1.1-1
[37] XML_3.98-1.11 pacman_0.4.6 ggplot2_2.2.1 tidyr_0.8.0 purrr_0.2.4 readr_1.1.1
[43] magrittr_1.5 codetools_0.2-15 scales_0.5.0 htmltools_0.3.6 assertthat_0.2.0 colorspace_1.3-2
[49] brew_1.0-6 stringi_1.1.7 visNetwork_2.0.3 lazyeval_0.2.1 munsell_0.4.3

6

Can you please be more specific on the crash log?

7

Just my whole computer crashes and R restarts. Do you know where I can find logs for this? thank you!
PS: If you have time: does the example work on your machine?
PPS: logloss is already defined in the package - correct? I don’t have to call the custom.metric part by myself - do I?

8

Sorry that for now I don’t have the environment to reproduce it. Can you please report it by submitting a new issue at here: https://github.com/apache/incubator-mxnet/issues ? There will be other engineers investigating.

In the issue, please at least re-paste the sessionInfo() shown above, and provide a complete and as-simple-as-possible script to reproduce the crash.

My straight guess would be that mx.nd.clip somehow doesn’t work, which crashes the program by feeding a negative number to mx.nd.log.

9

@hetong007 Thank you for the guess. I posted it here: https://github.com/apache/incubator-mxnet/issues/10739

Would have an idea for a workaround? If not then ok for now, thank you for your help!