Gelsight for Sim2Real

  1. What and why
  2. Previous works
  3. What makes it special
  4. Result

What and Why

This work explores simulating Gelsight so that agents can be trained in simulation and transferred to real hardware systems with minimal/no difficulties. This would require high fidelity simulation of the GelSight sensor.

Previous Works

My initial approach [1] in simulating Gelsight used raytracing techniques to recreate the 3d pointcloud of the contact surface geometry. In reality, a Poisson solver based surface reconstruction is used to reconstruct the depth image and normals from the gradient image.

What makes it special

This approach simulates the raw gradient RGB image coming from the sensor.


The following image shows a comparison of the simulated image and the image acquired from the real sensor on similar objects.

The top row of images are acquired from the hardware and the bottom one is the simulated one.

Here is a paper detailing the approach, and presented at ICRA 2019 ViTac Workshop.

Achu Wilson

Roboticist, still learning!