"I won't forget the look on Jacinda's face when she finally held the baby", Gayford said.
The Irish name means "radiant" or "bright".
The couple have explained why they settled on Neve Te Aroha Ardern Gayford.
"She looked absolutely stunned and just very, very happy".
"Te Aroha was our way of reflecting the amount of love this baby has been shown before she arrived and all of the names we were gifted along the way".
"We're not placing any great expectations on this little baby except for happiness and love", she said.
The last leader to give birth while holding office was late Pakistani Prime Minister Benazir Bhutto.
"To me, in these moments, in those messages [from people around the country], we found a reflection of the kindness of New Zealand people".
At almost three days old the tiny tot will make her first public appearance at 11am cradled in mother Jacinda Ardern's arms with proud dad Clarke Gayford.
Neve is Ardern's first child and she weighed in at 3.31kg when she made her way into the world at 4.45pm on Thursday 21 June.
Tottenham's Harry Kane shares what Mauricio Pochettino texted him on Monday
While some U.S. fans seemed happy with the idea of supporting England , others clearly weren't convinced by Corden's game plan. It is down to myself to put it right, and there is no better way to start with a win and a couple of goals.
Ms Ardern spoke briefly to reporters today before heading home, revealing her and her partner Clarke Gayford had named their daughter Neve Te Aroha Ardern Gayford.
Tweeting her congratulations to Ardern and Gayford earlier in the week, former NZ Prime Minister Helen Clark said their parenting arrangements were gender-equality in action.
All the presents for baby Neve would be passed on to another family once they had been used, Ms Ardern said.
Arden is the first elected world leader to take maternity leave, handing over the reigns to Deputy Prime Minister Winston Peters for six weeks, and only the second to have a child while in office - both roles she has spent months downplaying.
She will take six weeks of leave before returning to work.
"It's just something really lovely in a world where so much is unfair that we're all very excited about it", said Frances Gerbic, a music teacher living in Auckland.
"They congratulated the PM and Clarke and said they were looking forward to seeing them in the fall", the spokesperson said.
She added: "I hope for little girls and boys that there's a future where they can make choices about how they raise their family and what kind of career they have that are based on what they want and what makes them happy".