Rozwiązanie znalazłem na zagranicznym forum, ale i tak miałeś rację.
"Looking at your layer stack, you seem to have everything set up correctly. The only thing I can think that might cause this is that you're not using a complete black color when painting the mask. Try hitting the D key to make sure your foreground and background color are completely black and white. Then try painting the mask again."