Oke found the problem myself. The problem is related to this forum issue:
Basically the memory of the IO chip is limited and therefore you cant control 8 main channels separately (Maybe this can be solved in a smart way in the software, but it is also a hardware problem). Quite a limitation in my opinion. You still have the 6 FMU (AUX) ports, which you can use to control the airplane. However, this is not preferred because when the FMU fails these cannot be controller any more. Also I want to use all 14 pwm ports to use the full capability of my airplanes and to maximize my reliability.
FYI, the parameter #define PX4IO_MAX_MIXER_LENGHT only limits the length of a single mixer. So in my case this was not a limiting factor as each individual mixer is relatively simple. Might be nice to update the documentation about the custom mixing a little bit and include limitations like this.
This topic can be closed. Maybe even deleted, but might be nice for people to have a little bit more information about the problem.