Make sure that your alpha also has sufficient resolution. Higher resolution on your alpha will result in finer detail. Try at least 2048 x 2048 pixels. Alphas are usually white on black background.
You can apply the mask directly to your mesh without using polypaint. For example…
- Import your Alpha.
- You have two options as how to apply the alpha. You can use the DragRect stroke and draw it on the mesh. You can also go to “Tool > Masking > Mask By Alpha” and click “Mask By Alpha” button.
- Then use Inflate.
So you need a high resolution Alpha and high resolution mesh. If it is still rough then you will still need to subdivide your mesh some more.