I seem to have found a solution to this. When I examine check the status of the compass using Mavros and looking at the sent topic, it does not always send values. Maybe the sensor does not start correctly. If I restart the pixhawk it starts up and I'm able to recieve values from mavros.
When I can see that the sensor is publishing, the compass calibration is completed successfully. It is however strange that I am able to calibrate all the compass positions in the first place, since this would imply that the sensor is started and working.