Content retrieval

After testing with our real courses, we have decided to obtain a golden case of almost 5000 questions and answers https://eduplex.atlassian.net/browse/EDX-529 and proceed with more experiments https://eduplex.atlassian.net/browse/EDX-530

Experiments with this golden case and different LLMs do not show a big difference in the retrieved results

But there is a big difference between the results from the golden case (above 90%) and our test learning content (that stays below 40% of retrieved results success)

We decided to test different chunking strategies to try to improve the retrieval success scores.

After using the sentence splitter strategy we got improved results for our test learning content: