Okay so three things:
1. That's not a right triangle, it's isosceles. The formula depends on it being a right triangle. With an isosceles triangle the sides could be literally any length. That's why you're getting the exact same distance you put in.
2. You're using a different input. The height and angle are still the same, but you're using 17,188 feet instead of the agreed upon 15,840 feet.
3. My math with the 300 ft only used the height of the observer and the angle of 0.02 to find the distance at which the angle would be 0.02. It didn't use the three mile distance at all.
