This book constitutes the proceedings of the 5th International Workshop on Human Behavior Understanding, HBU 2014, held in Zurich, Switzerland, in September 2014. The 9 full papers presented in this volume were carefully reviewed and selected from 18 submissions. They are organized in topical sections named: social signals; face and affect; motion analysis; and multiparty interactions.