A 'gentlemanly agreement' means an informal, unwritten agreement based on trust and honour between two parties, with no legal backing.
Horace gets angry at this phrase because he realises he has been completely deceived. He had trusted the young woman and opened the safe for her, believing she was the lady of the house. Later, when he is caught by the real police, he understands she was herself a thief who tricked him using a false 'gentlemanly agreement' — while he kept his word, she did not. He feels cheated and humiliated.
Source: A Question of Trust, Chapter 4
---