tl;dr: 12awg is fine, your losses are going to be 1.6%, or 162W at maximum solar output under ideal conditions (less when there's less power generated), and the cost difference between 10awg and 12awg would probably take you 10+ years to break even with the power loss reduction, so it's probably not worth it to use 10awg.
The rest of this long post provides the manual way to figure this all out, and explains how the equations fit together so you can be confident when you use an online calculator, or double check their results with your inputs.
I have decided to go with a sunny boy 7.7 inverter on our grid tied array. We are using 30- 340 watt q cell panels and there will be 3 strings going to the inverter.
My only question is the calculator for voltage drop is showing only a 2% drop for the 170 feet of wire to the inverter using 12 gauge wire, does this sound right?
In my mind I would have thought larger was needed but I'm used to lower voltage AC runs.
It will be 3 strings back to the inverter, unless the other is more desirable?
Voltage drop is a matter of amperage and resistance. You have 10A per string, and 3 of those strings. Let's do the calculation for one string:
12awg wire has a resistance of 1.588 ohms per 1000 feet.
You have "170 feet of wire to the inverter" which I'm going to assume means you have 170 feet from the inverter to the panels, and actually have 340 feet of wire for each string - 170 feet going from the inverter to the panels, and 170 feet going from the panels to the inverter. Again, this is all for one string of panels. So you should have 6 wires total (aside from ground, communication, and other non-power cabling).
I'm going to use units from here on out in order to help others understand how these all relate to each other and fit together. I'll place parenthesis around the number and its unit to avoid confusion when complex units like ohm's per foot (ohms/ft) are used.
340 feet of 12 awg wire is (340ft) * ( (1.588ohm/kft) / (1000ft/kft) ) = (340ft) * (0.00158ohm/ft) = (0.5399ohm)
So your total resistance for one string is 0.54 ohm.
Voltage drop is calculated using Ohm's law: Voltage(V) = Current(A) * Resistance(ohm). So at your panel's maximum current, 10A, the voltage drop is:
(10A) * (0.54ohm) = (5.4V)
We could find the string's voltage, subtract the drop, and then figure out the loss, however since we know the current and voltage drop, we know how many watts are lost. Since we can easily figure out the total wattage of the array, we can use the maximum wattage and the lost wattage to figure out the energy loss.
Power can be calculated with Power(W) = Voltage(V) * Current(A):
(5.4V) * (10A) = (54W)
So your wiring between the inverter and the panels will consume 54 watts when your panels are at their maximum current of 10A. Since this consists of two runs of wire, that means each 170 foot length of wire will consume 27W of power. Nearly 100% of this will be converted into heat, so you effectively have (27W) / (170ft) = (0.16W/ft) of heating for each wire. This is too little to notice, though you do have 6 of these running next to each other, so your pipe will experience (6wires/pipe) * (0.16W/ft) = (0.95W/pipeft) of heating. A sensitive temperature camera could probably see this heating, but it's insignificant in electrical terms.
Ok, we're ready to calculate the total loss. With one string of 10 panels your maximum generated power is (10panels/string) * (340W/panel) = (3400W/string).
Now we can subtract the loss to find the maximum usable power at the inverter will be (3400W/string) - (54W/string) = (3346W/string).
The usable power, as a percentage of generated power is (3346w/string)/(3400W/string) = (98.4%) which means you are losing 1.6% of your maximum power output, and this loss will only occur at maximum amperage (10A) - so most of the time it'll be less.
You can also calculate the loss directly: (54W/string)/(3400W/string) = (1.6%).
Now, you might think that this loss is multipled by three to give you the total array loss, but you need to remember that the total power output also needs to be multiplied by three, so the (1.6%) loss will be the same no matter how many strings of the same type you use.
( (3string) * (54W/string) ) / ( (3string) * (3400W/string) ) = (162W) / (10200W) = (1.6%)
You loss in watts increases - 162W lost across all three arrays - and that's not a trivial amount. Over an ideal year in an ideal climate with an ideal setup you could be losing as much as (162W) * (6H/day) * (365.25day/year) = (355000WH/year), or 355KWH/year. If your electricity normally costs $0.15/KWH then that's a loss of $53/year.
So let's look at how this changes if you go from 12awg to 10awg, which has a resistance of (0.999ohm/kft). Since the resistance to wattage loss is a linear relationship, we can take a shortcut here by calculating the difference in resistance, then scaling the loss:
(0.999ohm/kft10awg) / (1.588ohm/kft12awg) = (0.63 12awg/10awg)
Now we can use that to find all our end values through multiplication:
(162W/12awg) * (0.63 12awg/10awg) = (102W/10awg) = 102W total loss
(1.6%/12awg) * (0.63 12awg/10awg) = (1%/10awg) = 1% loss
($53/year12awg) * (0.63 12awg/10awg) = ($33/year10awg) = $33/year cost
You can re-do all the above equations to confirm that the shortcut is indeed correct.
Let's go one step further, then. Assuming you haven't bought the wire, let's find out what the price difference is, and how long until you pay it off. You'll need 170 * 6 feet of wire, which is just over 1000ft:
1000ft of 12awg wire is $190
1000ft of 10awg wire is $390
So there's a $200 difference. The cost difference of the loss of the two wires is $53 - $33, or $20. So, under ideal conditions with 6 bright sunny hours directly on the panels every day, it would take you 10 years to make up the difference.
If you've already purchased the 12awg wire, then it'll take you just under 20 years, assuming you can't recoup the cost of the 12awg wire.
Given that conditions aren't going to be perfect, and that you electricity cost is possibly higher, I'd guess that the difference between going from 12awg to 10awg is not worth it. But put in your own values and find out.
Electrically and mechanically, 12awg is fine for your specific use. You could go as high as 18awg, and given the cost of copper maybe there's additional savings to be had despite the higher losses of the thinner wire, but 12awg is ubiquitous, and is pretty low cost relative to the amount of copper it contains, so it's likely that you wouldn't find significant savings with cheaper wire.