I just had a look at your rosbag, compliments on that it helped a lot to give me an idea on what is happening. From your description I understood that you considered that scenario to be a straight line without obstacles and you expected the drone to smoothly go straight to the goal. You can see in the rosbag, that the /local_pointcloud topic displays points. Therefore, the drone sees obstacles and will react to it, this scenario is therefore full of obstacles and the drone would never move straight.
First, you are flying too low, the drone constantly sees the ground. This is not an anticipated use case. You either need to set the goal_z_param higher (to fly higher) or set the min_box_z param lower to crop the point cloud more. You can display the cropping box in rviz using the topic /bounding_box. This box is used to crop the original pointcloud, and the algorithm will ignore all obstacles that lie outside. The ground is not allowed to lie in there.
Second, if you would expect the drone to move in between those two trees, you would need to adjust the bounding box size accordingly. To move in small spaces the parameter max_box_x and related need to be tuned. As an input, I would first try to avoid a single obstacle, passing between obstacles is harder and those might be too close together.
As an input, a video from our last outdoor test (PR #106, which btw is now merged into master):