It actually takes #4 to go 180' @50A /240V
#6 is only spec'd for 105' @50A /240V
You might even be better off using #2 and branching off from it with #6 to a couple of boxes, that way you have excess capability.
The voltage drop of using too small a gauge may cause you problems down the road. NEC only allows a maximum Voltage drop of 2% for single phase.
A good method to figure voltage drop is:
22 X wire length in feet X current in amps / circular mils.
the 22 figure is for copper only, if using aluminum, change it to 36.
Even though you told me all the electricians you spoke with said that #6 would be OK, in this case size does matter.
Be careful and good luck.
Dallas