Hi,
I cannot reproduce the Sonar classification example in the R tutorial:Develop a Neural Network with MXNet in Five Minutes. It behaves as though there is a memory bug in R or mxnet.
When I run the example, it classifies all the test samples into a single category. For example:
Loading required package: mlbench
...
Start training with 1 devices
[1] Train-accuracy=0.222222222222222
[2] Train-accuracy=0.476190476190476
[...
[20] Train-accuracy=0.476190476190476
Output from head(preds) is:
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12] [,13] [,14]
[1,] 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[2,] 1 1 1 1 1 1 1 1 1 1 1 1 1 1
...
[,99] [,100] [,101] [,102] [,103] [,104] [,105] [,106] [,107]
[1,] 1 1 1 1 1 1 1 1 1
[2,] 0 0 0 0 0 0 0 0 0
Final output is:
test.y
pred.label 0 1
1 60 47
(there is only one row here as all samples were classified into the one category)
I have run the code about 10 times and the output was always as shown above, except in one test where the classification behaved as expected (i.e. some samples classified as “0”, and some as “1”):
Loading required package: mlbench
...
Start training with 1 devices
[1] Train-accuracy=0.344444444444444
[2] Train-accuracy=0.352380952380952
...
[20] Train-accuracy=0.352380952380952
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12] [,13] [,14]
[1,] 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[2,] 0 0 0 0 0 0 0 0 0 0 0 0 0 0
...
[,99] [,100] [,101] [,102] [,103] [,104] [,105] [,106] [,107]
[1,] 1 0 1 0 0 1 1 0 0
[2,] 0 1 0 1 1 0 0 1 1
Final output is:
test.y
pred.label 0 1
0 26 39
1 34 8
(there are two rows here as some samples were classified into category “0” and others into “1”)
Given that the results are not consistent (ie. the output is not always the same), could this be a memory bug or some problem with my installation?
My code is exactly the same as in the tutorial (except for the head(preds) statement):
require(mlbench)
require(mxnet)
data(Sonar, package="mlbench")
Sonar[,61] = as.numeric(Sonar[,61])-1
train.ind = c(1:50, 100:150)
train.x = data.matrix(Sonar[train.ind, 1:60])
train.y = Sonar[train.ind, 61]
test.x = data.matrix(Sonar[-train.ind, 1:60])
test.y = Sonar[-train.ind, 61]
mx.set.seed(0)
model <- mx.mlp(train.x, train.y, hidden_node=10, out_node=2, out_activation="softmax",
num.round=20, array.batch.size=15, learning.rate=0.07, momentum=0.9,
eval.metric=mx.metric.accuracy)
preds = predict(model, test.x)
head(preds)
pred.label = max.col(t(preds))-1
table(pred.label, test.y)
I installed mxnet from source yesterday, checking out branch 0.12.0. The session info is:
R version 3.4.3 (2017-11-30)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Fedora 27 (Twenty Seven)
Matrix products: default
BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] methods stats graphics grDevices utils datasets base
other attached packages:
[1] mxnet_0.12.0 mlbench_2.1-1
loaded via a namespace (and not attached):
[1] Rcpp_0.12.14 pillar_1.0.1 compiler_3.4.3 RColorBrewer_1.1-2
[5] influenceR_0.1.0 plyr_1.8.4 bindr_0.1 viridis_0.4.0
[9] tools_3.4.3 digest_0.6.13 jsonlite_1.5 viridisLite_0.2.0
[13] gtable_0.2.0 tibble_1.4.1 rgexf_0.15.3 pkgconfig_2.0.1
[17] rlang_0.1.6 igraph_1.1.2 rstudioapi_0.7 bindrcpp_0.2
[21] gridExtra_2.3 downloader_0.4 DiagrammeR_0.9.2 dplyr_0.7.4
[25] stringr_1.2.0 htmlwidgets_0.9 hms_0.4.0 grid_3.4.3
[29] glue_1.2.0 R6_2.2.2 Rook_1.1-1 XML_3.98-1.9
[33] readr_1.1.1 purrr_0.2.4 tidyr_0.7.2 ggplot2_2.2.1
[37] magrittr_1.5 codetools_0.2-15 scales_0.5.0 htmltools_0.3.6
[41] assertthat_0.2.0 colorspace_1.3-2 brew_1.0-6 stringi_1.1.6
[45] visNetwork_2.0.3 lazyeval_0.2.1 munsell_0.4.3
Does anyone know what the problem could be?
BTW - I am new to R and mxnet so apologies if I have made a newbie error