zhRddlmZddlmZmZmZmZddlmZm Z m Z GddeZ y)) annotations)AnyListOptionalcast) TextSplitter Tokenizersplit_text_on_tokenscveZdZUdZ d d fd Z d dZd dZddZdZde d<dd Z xZ S)%SentenceTransformersTokenTextSplitterz8Splitting text to tokens using sentence model tokenizer.intc t|di|d|i ddlm}||_||j |_|j j|_|j|y#t$r t dwxYw)zCreate a new TextSplitter. chunk_overlapr)SentenceTransformerzCould not import sentence_transformers python package. This is needed in order to for SentenceTransformersTokenTextSplitter. Please install it with `pip install sentence-transformers`.)tokens_per_chunkN) super__init__sentence_transformersr ImportError model_name_model tokenizer_initialize_chunk_configuration)selfrrrkwargsr __class__s h/var/www/html/engine/venv/lib/python3.12/site-packages/langchain_text_splitters/sentence_transformers.pyrz.SentenceTransformersTokenTextSplitter.__init__ s ?6??  A%)$//: .. ,,>N,O N  s A((A=c $tt|jj|_||j|_n||_|j |jkDr3t d|jd|jd|j dy)NzThe token limit of the models 'z' is: z. Argument tokens_per_chunk=z > maximum token limit.)rr rmax_seq_lengthmaximum_tokens_per_chunkr ValueErrorr)rrs rrzESentenceTransformersTokenTextSplitter._initialize_chunk_configuration#s)-S$++2L2L(M%  #$($A$AD !$4D !  4#@#@ @1$//1BC5567..2.C.C-D)+  Acdfd }tjjjj|}t ||S)aSplits the input text into smaller components by splitting text on tokens. This method encodes the input text using a private `_encode` method, then strips the start and stop token IDs from the encoded result. It returns the processed segments as a list of strings. Args: text (str): The input text to be split. Returns: List[str]: A list of string components derived from the input text after encoding and processing. c,j|ddS)N)_encode)textrs r%encode_strip_start_and_stop_token_idsz_SentenceTransformersTokenTextSplitter.split_text..encode_strip_start_and_stop_token_idsDs<<%a+ +r#)rrdecodeencode)r)rr)strreturnz List[int])r _chunk_overlaprrr+r )rr)r*rs` r split_textz0SentenceTransformersTokenTextSplitter.split_text5sE ,--!22>>((8  $CCr#c6t|j|S)ayCounts the number of tokens in the given text. This method encodes the input text using a private `_encode` method and calculates the total number of tokens in the encoded result. Args: text (str): The input text for which the token count is calculated. Returns: int: The number of tokens in the encoded text. )lenr()rr)s r count_tokensz2SentenceTransformersTokenTextSplitter.count_tokensPs4<<%&&r#l _max_length_equal_32_bit_integercV|jj||jd}|S)Ndo_not_truncate) max_length truncation)rr,r5)rr)&token_ids_with_start_and_end_token_idss rr(z-SentenceTransformersTokenTextSplitter._encode`s4151F1F <<(2G2 . 65r#)2z'sentence-transformers/all-mpnet-base-v2N) rr rr.r Optional[int]rrr/None)rr<r/r=)r)r.r/z List[str])r)r.r/r r-) __name__ __module__ __qualname____doc__rrr1r4r5__annotations__r( __classcell__)rs@rr r s~B C*. PPP( P  P  P0#0 $D6 '-2$c16r#r N) __future__rtypingrrrrlangchain_text_splitters.baserr r r rr#rrGs"",,WW^6L^6r#