AI spells better times ahead for the subject matter experts


It is commonly believed that AI will empower the employees who are willing to learn – others will find it hard to keep their jobs. But there’s an even bigger opportunity for the subject matter experts. They will not only learn the changes being brought about by AI , but permanently enrich their jobs by becoming the trainers and administrators of the AI apps.

Firstly, let us understand why a human being – no matter whether she is a subject matter expert or not,  is needed to train the modern AI systems.

Human-in-the-loop systems

Humans will be instrumental in building what are known as human-in-the-loop systems.

As per this article by Mothi Venkatesh human-in-the-loop system combines “Supervised Machine Learning” and “Active Learning”. I am reproducing the definition from the article.


Supervised ML, curated (labelled) data sets used by ML experts to train algorithms by adjusting parameters, in order to make accurate predictions for incoming data.

Active Learning, the data is taken, trained, tuned, tested and more data is fed back into the algorithm to make it smarter, more confident, and more accurate. This approach–especially feeding data back into a classifier is called active learning.

It is obvious that we need a human being where the confidence level in the algorithm is low due to insufficient training data or the risk associated with an inaccurate inference is very high. But why would we need a subject matter expert. They are not just another “human” augmenting the algorithm’s artificial intelligence.

Subject matter experts as trainers and administrators of AI

This article by Daniel Faggella explains the need for subject matter experts. I am reproducing three ways as written in the article.

  • Determining the Business Problem an AI Solution Can Solve. For example, let’s say an insurance company wants to become more competitive. A high-level employee at the company, who has been in the industry for more than 20 years, may know from experience that clients often complain about the time it takes to process claims.
  • Determining How AI Can Solve That Business Problem. For example, a marketer might tell a data scientist to make sure the model they build can predict the best price to bid for ad space on Google ad Network without realizing that such an application is nascent in practice and technically challenging. A data scientist would be able to explain this to the marketer, preventing them from promising their team and stakeholders too much.
  • Maintaining and Updating the System Once It’s Built. Take for example a manufacturing company with a large number of heavy machinery to maintain. The subject-matter experts collaborate with data scientists to develop an AI system that gathers data from IoT sensors on the machines to predict the optimal time to do preventive maintenance on them.


To understand this better we need to take a closer look at how the AI apps work.

  1. Every AI app needs training data that needs to be prepared by a human expert. E.g. we need to have a lot of data about the films and the people who watch those films to intelligently recommend a bunch of films to a new user.
  2. After preparing the training data a data scientist could use this data to train and test an algorithm. He will repeat this step till he finds a suitable algorithm to generate a model that can predict or infer as precisely as the human expert who has prepared the training data in step 1. This step could take several iterations.
  3. Once the algorithm which can most precisely predict , the same can be used to create a model. This model is handed over to the subject matter expert to be used for predictions or inferences.
  4. The SME uses this model for day to day predictions and inferences. He applies his own judgement to decide if the predictions and inferences are precise enough by comparing what the algorithm has predicted or inferred with what the SME would have done.
  5. If the model is not accurately inferring or predicting the SME would override the algorithm and manually assign predicted or inferred values.
  6. This edited data is added as new input data and the same is used to retrain the algorithm. This retraining will ensure better results going forward. Step 4 to 6 are repeated by the SME with no intervention from the data scientist. The SME would really make the application more and more precise. As it becomes more precise it will more closely mimic the expert’s inferences and predictions.

Who can be a subject matter expert.

You don’t have to be highly placed or academically well qualified. There’s nothing better than years of expertise. E.g. a security guard knows how to spot an intruder in a crowd. He can help a data scientist build an automated threat detection system.

Thank You!

Please share your views and suggestion:)

I hope you enjoyed this article. Don’t forget to share your views. You can also tweet your comments to @Rezoomex and @vinayakj with #TechHiringBlog

VinayakSir_ProfilePicVinayak Joglekar is a technology & recruitment expert with around 40 yrs. of experience. He is a distinguished thought leader who has changed trends in recruitment and futuristic software development. He is acclaimed for recruiting and developing great talents in IT.

Vinayak believes in empowering IT recruiters with technical knowledge so that they can earn their due respect. Currently, he is the CTO of Synerzip and Director of Rezoomex.


Leave a Reply