The software provider questioned more than 220 financial advisers and found client communication was the most important thing to keep control of.
Other factors that firms were unwilling to outsource were client reporting, portfolio monitoring and conducting attitude to risk questionnaires.
This was despite 67 per cent of firms having less than five employees, showing even the smallest firms want to do all the work themselves.
The part of the service they were most likely to outsource was the asset allocation model with 36 per cent of advisers saying they would outsource this.
{desktop}{/desktop}{mobile}{/mobile}
The reasons for this were the need for external expertise, need for additional resources and to save costs.
Rob Gleeson, head of research at FE, said: "This survey demonstrates that the relationship between advisers and their clients remains paramount. Providers of outsourced solutions, such as model portfolios, need to acknowledge that IFAs want to remain in the driving seat and to maintain direct contact with their clients.
"This is as it should be, as IFAs know their clients far better than third-party fund managers, and have a deeper understanding of their financial goals and capacity for risk."

• Want to receive a free weekly summary of the best news stories from our website? Just go to home page and submit your name and email address. If you are already logged in you will need to log out to see the e-newsletter sign up. You can then log in again.