Why AI Isn’t Reliable for CII Exam Preparation

In the financial services industry, the integration of AI technology has garnered attention as a potential game-changer. With promises of efficiency and accessibility, many turn to AI platforms like Chat GPT for assistance, particularly in exam preparation. However, as we delve deeper into its efficacy, concerns arise regarding the accuracy and reliability of AI-generated content, prompting a critical examination of its suitability for CII exam preparation.

This article is correct as at 9 April 2024.

According to Professional Adviser, 32% of financial advice firms are using AI, and as many as 29% have implemented this within their working practices to help with things like writing emails or research. As we all know, the advances in AI have been off the scale in the last few years and what was once a sci-fi dream, is now a part of everyday life. Not only are companies using AI, but individuals are also starting to use it, especially with the introduction of free open-source AI platforms such as Chat GPT.

There is a temptation to use these platforms to help with exam revision and technical training for key financial advice concepts. We know this because we have heard of candidates using Chat GPT to generate mock exam questions, for example. Whilst on the face of it this sounds like a great idea, when we tested it, the reality was not so great.

We tested to see how the mock questions we generated on Chat GPT might compare to those written by our team of experts.

We asked Chat GPT “Create 5 multiple-choice questions for me which would test my knowledge of the CII R03 exam syllabus”. This returned 5 questions, but the first problem we encountered is that they were all about pensions, which though mentioned in R03, isn’t the core of the exam. The first question was as follows:

Question 1

When comparing defined benefit (DB) and defined contribution (DC) pension schemes, which statement is true?

1. DB schemes typically provide a guaranteed income based on contributions and investment growth.
2. DC schemes promise a fixed income throughout retirement, unaffected by market fluctuations.
3. DB schemes offer flexibility in contributions and investment choices.
4. DC schemes offer a predetermined income based on salary history and years of service.

Answer 1: A

The correct answer was given as option A), which isn’t correct. DB schemes don’t typically provide a guaranteed pension based on contributions and investment growth, it’s based on salary and length of service. In fact, none of the answer options is correct, the system has clearly mixed-up DB and DC schemes in generating the answer options.

So we thought we’d get more specific to see if that helps. We asked “Create 5 mock multiple-choice questions for me which would test my knowledge of the CII R03 exam syllabus, specifically income tax for the 23/24 tax year”; we hoped it would narrow down the topic. In this example, the results were worse – 4 of the 5 questions generated were wrong, and the fifth was too easy to be worthwhile. Some questions returned were relevant to the 22/23 tax year. Another example of a generated question is shown below:

Question 5

What happens if an individual exceeds their annual tax-free Personal Allowance?

1. They will be taxed at a higher rate for all income earned.
2. They lose their entitlement to state benefits.
3. They must pay a penalty fee.
4. They are subject to higher National Insurance contributions.

Answer 5: The correct answer is: A) They will be taxed at a higher rate for all income earned.

Explanation: If an individual exceeds their annual tax-free Personal Allowance, the excess income is subject to taxation at the applicable tax rates for the additional income earned. This means they will be taxed at a higher rate on the income exceeding their allowance.

This is incorrect because after the tax-free personal allowance, the basic rate of tax would be applied until the basic rate band was exceeded, then the higher rate would apply. Perhaps what has happened here is that the system is using the word ‘higher’ to mean a higher than 0%, i.e., 20%, which isn’t technically incorrect, but because the word ‘higher’ is used, it implies the 40% higher rate of tax. This is a little muddled, and whilst it might be more obvious to those who know, to someone who is learning this for the first time, we can see why this would be confusing.

Though this was amusing for us to read, the consequences of relying on this as a resource to get you through an exam could be severe. That’s not to say that it couldn’t be helpful – perhaps asking something like ‘Explain how to calculate Sharpe Ratio’ might return a useful answer, but the problem is that if you’re not familiar with the subject already, you won’t know if the answer that’s returned to you is correct or not.

All of our exam resources go through a three-point check when they are created, and then they’re technically reviewed and updated by at least two technical experts each year (sometimes four, depending on the topic!), with additional technical checks and updates outside of our major update periods. We have a team of experts with over 120 years of combined experience and technical expertise in their chosen subjects. Some also write for the CII, so you can know that all our resources are reliable and accurate.