Found other threads here with a similar problem. The recommended solution is to roll back to U14. If you use process explorer you will see Microsoft.Crm.Application.Outlook.WebFormsHost running repeatedly. Search based on that term to see them.
I uninstalled the 2013 client, installed 2011 applied U15 and the problem started again. Uninstalled 15 and applied 14 and there is no problem.
I have another machine running 64bit Office and it doesn't have this problem with 15, I don't know if that is significant.