Enabling Bidirectional DShot

As I mentioned - set DSHOT_BIDIR_EN to enable.

What PX4 version are you running?

You mentioned custom H7 build - does it follow any Pixhawk standard? FMUx?
Is bdshot compiled into your firmware?