Smallest AWG that keeps voltage drop within your limit and stays within ampacity ratings.
Use 8 AWG (8.37 mm²) for this circuit.
Next stepInstall the wire with proper crimp or lug terminations rated for the AWG and apply heat-shrink tubing over all connections.