> For the complete documentation index, see [llms.txt](https://docs.eduplex.eu/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.eduplex.eu/text-analysis/ai-tutor-rag-system/retrieval-performance-indexing-pdf-vs-xapi-and-keywords-vs-questions.md).

# Retrieval Performance Indexing pdf vs xapi, and Keywords vs Questions

**Goal**\
Comparing retrieval performance across two indexing methods (pdf and xapi) and two query methods (Keywords vs Questions)\
\
**Data**\
Future Skills Module with 20 test questions.\
Keywords generated from questions using GPT-4 (gpt-4-1106-preview). Indexed documents in OpenSearch from pdf and xapi structure.\
\
**Method/Approach**\
Retrieval evaluation using two different indexing formats in OpenSearch (pdf and xapi) and two query methods (keywords vs full question). Vectoization with sbert. Relevance evaluated using gpt-4 as relevance grader (prompt based on trulens-eval library) to score similarity between query and retrieved chunks.\
\
**Results**\
Average relevance score for xapi index: 0.45. Average relevance score for pdf index: 0.3\
Full questions as queries resulted in a higher average relevance score (0.47) compared to keywords (0.34).\
\
**Evaluation Metrics**\
Mean Context Relevance Score (LLM-based): score from 0 to 1, averaged over 2 runs for each query.\
\
**Conclusions**\
Indexing based on xapi structure resulted in better retrieval relevance scores compared to pdf indexing.Using full questions as queries provides more accurate retrieval compared to keywords. So question-based retrieval is preferable for higher context relevance.