How to find which part of body the keypoint relates to in Pose estimation?

Gluon
1

Hi, I am using gluoncv for Pose Estimation of Person and I am getting key points after the prediction but how do I check which keypoint is there for which body part. Like for example if I want to find the key point of Nose for the person in the image, how can I do that. Please help.

img0001_detected
img0001_detected300×666 45.2 KB

Predicted Keypoints:

[[ 37.52469 25.09375 ] [ 38.741013 20.53125 ] [ 35.09204 20.53125 ] [ 41.173668 25.09375 ] [ 30.226744 25.09375 ] [ 43.606316 38.78125 ] [ 25.361444 38.78125 ] [ 48.47162 45.625 ] [ 13.198198 57.03125 ] [ 49.687943 52.46875 ] [ 21.71247 47.90625 ] [ 39.957344 75.28125 ] [ 27.794096 75.28125 ] [ 38.741013 100.375 ] [ 29.01042 107.21875 ] [ 39.957344 120.90625 ] [ 27.794096 132.3125 ]]

2 
from gluoncv import data
print(data.mscoco.keypoints.COCOKeyPoints.KEYPOINTS)
3

How can I superimpose predicted points on image using gluonCV to check if those predicted points are properly estimated or not?

4

This is a simple example.

from gluoncv import model_zoo, data, utils

from gluoncv.data.transforms.pose import detector_to_simple_pose, heatmap_to_coord

detector = model_zoo.get_model('yolo3_mobilenet1.0_coco', pretrained=True, root='./model')

pose_net = model_zoo.get_model('simple_pose_resnet18_v1b', pretrained=True, root='./model')

detector.reset_class(["person"], reuse_weights=['person'])

url = 'https://raw.githubusercontent.com/dmlc/web-data/master/gluoncv/segmentation/mhpv1_examples/1.jpg'

filename = 'sample.jpg'

utils.download(url, filename)

x, img = data.transforms.presets.ssd.load_test(filename, short=512)

class_IDs, scores, bounding_boxs = detector(x)

pose_input, upscale_bbox = detector_to_simple_pose(img, class_IDs, scores, bounding_boxs)

predicted_heatmap = pose_net(pose_input)

pred_coords, confidence = heatmap_to_coord(predicted_heatmap, upscale_bbox)

from PIL import Image, ImageDraw

pil_image = Image.fromarray(img)

draw = ImageDraw.Draw(pil_image)

keypoints = data.mscoco.keypoints.COCOKeyPoints.KEYPOINTS

for keypoint_id in range(len(keypoints)):

    pred = pred_coords[:,keypoint_id,:]

    for i in range(pred.shape[0]):

        if (confidence[i,keypoint_id,:] > 0.2) == 1:

            draw.text(pred[i,:].asnumpy(),text=keypoints[keypoint_id], fill='red')

pil_image.show()
1 Like
5

Hi @dai-ichiro I really appreciate your help regarding my problem. But what my actual concern is that suppose if I have my own predicted keypoints then how I can use them as pred_coords and superimpose it on image to see how the keypoints appear according to those predicted keypoint?

6

If you prepare dummy data, you can use utils.viz.plot_keypoints .
This is a simple example.

from matplotlib import pyplot as plt
import mxnet as mx
from gluoncv import model_zoo, data, utils
url = 'https://raw.githubusercontent.com/dmlc/web-data/master/gluoncv/segmentation/mhpv1_examples/1.jpg'
filename = 'sample.jpg'
utils.download(url, filename)
x, img = data.transforms.presets.ssd.load_test(filename, short=512)
#your predicted keypoints
Predicted_Keypoints=mx.nd.array(
    [[144.49815, 188.80566],
     [155.40056, 175.33923],
     [133.59575, 182.07245],
     [177.2053 , 182.07245],
     [122.69336, 188.80566],
     [220.81493, 249.4046 ],
     [106.33977, 276.33746],
     [258.97333, 336.9364 ],
     [ 79.08377, 350.40283],
     [269.87573, 444.66785],
     [ 62.73018, 404.26855],
     [226.2661 , 437.93463],
     [160.85173, 444.66785],
     [237.1685 , 505.26678],
     [171.75414, 505.26678],
     [280.77808, 505.26678],
     [253.52211, 471.6007 ]])
Predicted_Keypoints = Predicted_Keypoints.expand_dims(0)
#dummy data
class_IDs = mx.nd.zeros(shape=(1,100,1))
confidence = mx.nd.ones(shape=(1,17,1))
bounding_boxs = mx.nd.zeros(shape=(1,100,4))
scores = mx.nd.zeros(shape=(1,100,1))
ax = utils.viz.plot_keypoints(img, Predicted_Keypoints, confidence,
                              class_IDs, bounding_boxs, scores,
                              box_thresh=0.5, keypoint_thresh=0.2)
plt.show()

Hope this helps.