Convert Batch documents to Embeddings

convert_batch_documents_to_embeddings converts whole batch of documents in vector embeddings at once

Usage

convert_batch_documents_to_embeddings(
  ollama_connection,
  document_path_list,
  splitter = "paragraph",
  model = "bge-large"
)

Arguments

ollama_connection

a connector object to an ollama instance to provide the embeddings.

document_path_list

a list containing full paths to files that need to be converted to embeddings.

splitter

the splitter type to use when preparing embeddings. Defaults to "paragraph". "sentence" can also be used.

model

the model to use for creating the embeddings. Defaults to "bge-large"

Details

Lets you convert as many documents as necessary by targeting a path list of documents, which can be easily generated by functions or manually. It returns a dataframe in the end that can be used to provide context to a LLM workflow.

Examples

conn <- get_ollama_connection()

document <- "Standing proudly on the Île de la Cité in the heart of Paris, 
France's capital city, lies one of the world's most beloved and historic 
landmarks: the magnificent Notre Dame Cathedral. This Gothic masterpiece 
has been welcoming pilgrims and tourists alike for over 850 years, since its 
construction began in 1163 under King Louis VII. With its towering spires, 
stunning stained glass windows, and intricate stone carvings, this beautiful 
church is a testament to medieval architecture and engineering skill. 
Unfortunately, a devastating fire ravaged the cathedral on April 15, 2019, 
but thanks to swift action from firefighters and restoration efforts 
underway, Notre Dame continues to inspire awe in those who visit her."

writeLines(document, con = "doc1.txt")

convert_batch_documents_to_embeddings(ollama_connection = conn, 
document_path_list = list("doc1.txt"))
#>      embeddings
#>          <AsIs>
#> 1: -0.02001....
#>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         text
#>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       <char>
#> 1: Standing proudly on the Île de la Cité in the heart of Paris,  France's capital city, lies one of the world's most beloved and historic  landmarks: the magnificent Notre Dame Cathedral. This Gothic masterpiece  has been welcoming pilgrims and tourists alike for over 850 years, since its  construction began in 1163 under King Louis VII. With its towering spires,  stunning stained glass windows, and intricate stone carvings, this beautiful  church is a testament to medieval architecture and engineering skill.  Unfortunately, a devastating fire ravaged the cathedral on April 15, 2019,  but thanks to swift action from firefighters and restoration efforts  underway, Notre Dame continues to inspire awe in those who visit her.