Since these are retreats, I presume Honor & Fortune tokens are lost. If the unit contains a leader, I presume an extra H&F token is lost.
I also presume that each die is considered separately, so if 3 Red were rolled, a single Red unit could be retreated 3 hexes. This isn't clear, so for example if 3 Green were rolled, but only 2 Green units were in play, would only 2 units be counted, or must 1 unit retreat 2 hexes. If each die is resolved separately, this problem is simply resolved.
“I dreamt of worldly success once.”
― Miyamoto Musashi