I needed a consent for a minor form notorized from my son's father to get his passport.the form can be downloaded from passport website..
www.travel.state.gov Click on passports tab and follow instructions from there.
He (the father)didnt have to accompany us to the passport office as long as I had this.
Also, get a permisssion to take him out of the country letter notorized from the absent parent unless you have proof of sole custody with you for customs etc.
Why take a chance?