0hpmdZddlZddlZddlZddlZddlZddlZddl m cm Z ddl mZddlmZedg ddZ ddZd Zd Zd Zd Zd ZdZdZ ddZdZdZdZdZdZy)z&Keras image dataset loading utilities.N)io_utils) keras_exportzkeras.utils.split_dataset)v1c(t|}|tjjtt t jfvrtd|dt|| | tdt||}|rK|tjdtd}tj|tj |t#|}t%|||\}}t |d|}t || d} t'|||}t'| ||} tjjj)|}tjjj)| } |tjjur:t+|r/t-|} | "|j/| }| j/| } |j1tjj2}| j1tjj2} || fS)aSplit a dataset into a left half and a right half (e.g. train / test). Args: dataset: A `tf.data.Dataset` object, or a list/tuple of arrays with the same length. left_size: If float (in the range `[0, 1]`), it signifies the fraction of the data to pack in the left dataset. If integer, it signifies the number of samples to pack in the left dataset. If `None`, it uses the complement to `right_size`. Defaults to `None`. right_size: If float (in the range `[0, 1]`), it signifies the fraction of the data to pack in the right dataset. If integer, it signifies the number of samples to pack in the right dataset. If `None`, it uses the complement to `left_size`. Defaults to `None`. shuffle: Boolean, whether to shuffle the data before splitting it. seed: A random seed for shuffling. Returns: A tuple of two `tf.data.Dataset` objects: the left and right splits. Example: >>> data = np.random.random(size=(1000, 4)) >>> left_ds, right_ds = tf.keras.utils.split_dataset(data, left_size=0.8) >>> int(left_ds.cardinality()) 800 >>> int(right_ds.cardinality()) 200 znThe `dataset` argument must be either a `tf.data.Dataset` object or a list/tuple of arrays. Received: dataset=z of type NzoAt least one of the `left_size` or `right_size` must be specified. Received: left_size=None and right_size=Noner.A)_get_type_spectfdataDatasetlisttuplenpndarray TypeErrortype ValueError_convert_dataset_to_listrandomrandintintseedshufflelen_rescale_dataset_split_sizes_restore_dataset_from_listfrom_tensor_slices is_batchedget_batch_sizebatchprefetchAUTOTUNE) dataset left_size right_sizerrdataset_type_specdataset_as_list total_length left_split right_split batch_sizes Z/var/www/html/engine/venv/lib/python3.12/site-packages/tf_keras/src/utils/dataset_utils.py split_datasetr, sB'w/$rzz JJ !!( 4=/ C  i/   /w8IJO <>>!SX.D D''L8:|Izojy12J {|45K+%wJ-&K33J?J''//44[AKBGGOO+ 70C#G,  !#))*5J%++J7K$$RWW%5%56J&&rww'7'78K { ""cRt||}g}tj}t||||D]I}|ttfvr( t j |}|j|9|j|K|S#t$rt j |t}YMwxYw)aConvert `tf.data.Dataset` object or list/tuple of NumPy arrays to a list. Args: dataset : A `tf.data.Dataset` object or a list/tuple of arrays. dataset_type_spec : the type of the dataset data_size_warning_flag (bool, optional): If set to True, a warning will be issued if the dataset takes longer than 10 seconds to iterate. Defaults to `True`. ensure_shape_similarity (bool, optional): If set to True, the shape of the first sample will be used to validate the shape of rest of the samples. Defaults to `True`. Returns: List: A list of tuples/NumPy arrays. dtype) _get_data_iterator_from_datasettime_get_next_sampler r rarrayrobjectappend) r"r%data_size_warning_flagensure_shape_similaritydataset_iteratorr& start_timesamplearrs r+rrxs*7"OJ" +  - 5hhv&  " "3 '  " "6 *+"   5hhvV4 5sA??$B&%B&c f|tk(rt|dk(r tdt|dtj ur}|dj }t|D]_\}}t j|j d|dk7s/td|d|dt j|j dntdt|dtt|S|tk(rt|dk(r tdt|dtj ur}|dj }t|D]_\}}t j|j d|dk7s/td |d|dt j|j d ntd t|dtt|S|tjjk(r&t!|r|j#}t|S|tj k(r t|Sy ) a2Get the iterator from a dataset. Args: dataset : A `tf.data.Dataset` object or a list/tuple of arrays. dataset_type_spec : the type of the dataset Raises: ValueError: - If the dataset is empty. - If the dataset is not a `tf.data.Dataset` object or a list/tuple of arrays. - If the dataset is a list/tuple of arrays and the length of the list/tuple is not equal to the number Returns: iterator: An `iterator` object. rzJReceived an empty list dataset. Please provide a non-empty list of arrays.zPReceived a list of NumPy arrays with different lengths. Mismatch found at index z, Expected shape=z Received shape=z<.Please provide a list of NumPy arrays with the same length.z5Expected a list of `numpy.ndarray` objects,Received: zJReceived an empty list dataset.Please provide a non-empty tuple of arrays.zQReceived a tuple of NumPy arrays with different lengths. Mismatch found at index z=.Please provide a tuple of NumPy arrays with the same length.z7Expected a tuple of `numpy.ndarray` objects, Received: N)r rrrrrshape enumerater4riterzipr r r r runbatch)r"r%expected_shapeielements r+r1r1sW$D w<1 =  '!* % 3$QZ--N'0  788G$**1-1BB$<<=3?**8)9:**,((7*;*A*A)BC++ !'!*-.0  CM"" e # w<1 >  '!* % 3$QZ--N'0  788G$**1-1BB$<<=3?**8)9:**,((7*;*A*A)BC++ !'!*-.0  CM"" bggoo - g oo'GG} bjj (G} )r-c #K t|}t|}t|tjt j fr t j|j}nd}d}|t|D]\}}|rq|t j|jk7rOtdt j|jdt j|jd|d|rM|dzd k(rEtj}t||z dkDr |rtjd t d d}|y#t$r tdwxYww) a "Yield data samples from the `dataset_iterator`. Args: dataset_iterator : An `iterator` object. ensure_shape_similarity (bool, optional): If set to True, the shape of the first sample will be used to validate the shape of rest of the samples. Defaults to `True`. data_size_warning_flag (bool, optional): If set to True, a warning will be issued if the dataset takes longer than 10 seconds to iterate. Defaults to `True`. start_time (float): the start time of the dataset iteration. this is used only if `data_size_warning_flag` is set to true. Raises: ValueError: - If the dataset is empty. - If `ensure_shape_similarity` is set to True and the shape of the first sample is not equal to the shape of atleast one of the rest of the samples. Yields: data_sample: A tuple/list of numpy arrays. NFz|Received an empty Dataset. `dataset` must be a non-empty list/tuple of `numpy.ndarray` objects or `tf.data.Dataset` objects.z StopIterationrr?r2rwarningswarnResourceWarning) r9r8r7r: first_samplefirst_sample_shaperDr;cur_times r+r3r3s]8  01,- lRYY $; jc "2v{99;x*,-27MMM3 "1..3* 1   ,   sEA(E,CEEEct|ttfvrtdt|DS|tjj k(rtt |jtrBi}|D]9}|jD]$\}}||vr|g||<||j|&;|Stdt|DS|S)z,Restore the dataset from the list of arrays.c3FK|]}tj|ywNrr4.0r;s r+ z-_restore_dataset_from_list..AsJ&RXXf%J!c3FK|]}tj|ywrWrXrYs r+r[z-_restore_dataset_from_list..MsNf&)Nr\) r r rAr r r rL element_specdictitemsr6)r&r%original_datasetrestored_datasetdkvs r+rr<sUDM)JC4IJJJ bggoo - &33T :! $ 6GGI6DAq 00/0c(+(+2215 6 6 $ #N_8MNN N r-c t|}t|}|/|ttfvr!||ttfvrtd|d|||ttfvrtd|d||ttfvrtd|d|dk(r|dk(r t d|tk(r |dks||k\s|tk(r|dks|d k\rt d |d ||tk(r |dks||k\s|tk(r|dks|d k\rt d |d |||cxk(r tk(rnn||zd kDr t d|tk(rt ||z}n|tk(r t|}|tk(rt ||z}n|tk(r t|}|||z }n|||z }||z|kDrt d||zd||df|dffD]"\}}|dk(s t d|d|d|d|d t|t|}}||fS)arRescale the dataset split sizes. We want to ensure that the sum of the split sizes is equal to the total length of the dataset. Args: left_size : The size of the left dataset split. right_size : The size of the right dataset split. total_length : The total length of the dataset. Raises: TypeError: - If `left_size` or `right_size` is not an integer or float. ValueError: - If `left_size` or `right_size` is negative or greater than 1 or greater than `total_length`. Returns: tuple: A tuple of rescaled left_size and right_size zjInvalid `left_size` and `right_size` Types. Expected: integer or float or None, Received: type(left_size)=z and type(right_size)=zTInvalid `left_size` Type. Expected: int or float or None, Received: type(left_size)=z. zUInvalid `right_size` Type. Expected: int or float or None,Received: type(right_size)=rGrz]Both `left_size` and `right_size` are zero. At least one of the split sizes must be non-zero.z=`left_size` should be either a positive integer smaller than z<, or a float within the range `[0, 1]`. Received: left_size=zB`right_size` should be either a positive integer and smaller than z< or a float within the range `[0, 1]`. Received: right_size=z^The sum of `left_size` and `right_size` is greater than 1. It must be less than or equal to 1.zvThe sum of `left_size` and `right_size` should be smaller than the {total_length}. Received: left_size + right_size = zand total_length = leftrightzWith `dataset` of length=z, `left_size`=z and `right_size`=z .Resulting zN side dataset split will be empty. Adjust any of the aforementioned parameters)rrfloatrrround)r#r$r'left_size_typeright_size_typesplitsides r+rrQs&)_N:&O .e "D?3,#F C4_4E G  U|!C ))7(8 =  /#u"E **9):! =  A~*/ @   # !^yL8 U " !^yA~ (>*>k    3 1_ l : e # 1_ a ,~.?l    >2U2  "Q & :  )l23 3 )$ %: 45 C :&  :-  !I- : , 22;J2F1G!, 1  #F+j'-BC t A:+L>+/ |<!F#>>  NC OzI j  r-cXt|trtSt|trtSt|tjrtjSt|t rt St|t jjrt jjSy)z!Get the type spec of the dataset.N) rLr r rrr_r r r )r"s r+rrsj'5! GT " GRZZ (zz GT " GRWW__ -wwr-ct|dS)z, "Check if the `tf.data.Dataset` is batched. _batch_size)hasattr tf_datasets r+rrs :} --r-c2t|r |jSy)z"Get the batch size of the dataset.N)rrrrts r+rrs*%%%r-c |dk7rdg}|}ng}ttjjj D]}tjjj tjjj |sU|jdrg|jdr|dd}|j||s|}n(t|t|k7rtd|d|tt|tt|} t j"j%} g} g} fd |DD]+} | j| j't(| | ||f-g}| D]+}|j+\}}|j|| |z } -|d vrSt|t| k7r'td t|d t| d dtt|}nId}t-j.t| fd}|D]!}||||t|z|t|z }#|#t1j2dt| dn.t1j2dt| dt|d| j5| j | Dcgc],}tjjj |.}}|r|t,j6j9d}t,j6j;|}|j=|t,j6j;|}|j=||||fScc}w)aMake list of all files in `directory`, with their labels. Args: directory: Directory where the data is located. If `labels` is "inferred", it should contain subdirectories, each containing files for a class. Otherwise, the directory structure is ignored. labels: Either "inferred" (labels are generated from the directory structure), None (no labels), or a list/tuple of integer labels of the same size as the number of valid files found in the directory. Labels should be sorted according to the alphanumeric order of the image file paths (obtained via `os.walk(directory)` in Python). formats: Allowlist of file extensions to index (e.g. ".jpg", ".txt"). class_names: Only valid if "labels" is "inferred". This is the explicit list of class names (must match names of subdirectories). Used to control the order of the classes (otherwise alphanumerical order is used). shuffle: Whether to shuffle the data. Default: True. If set to False, sorts the data in alphanumeric order. seed: Optional random seed for shuffling. follow_links: Whether to visits subdirectories pointed to by symlinks. Returns: tuple (file_paths, labels, class_names). file_paths: list of file paths (strings). labels: list of matching integer labels (same length as file_paths) class_names: names of the classes corresponding to these labels, in order. inferredrG/NzjThe `class_names` passed did not match the names of the subdirectories of the target directory. Expected: z, but received: c3rK|].}tjjj|0ywrW)r iogfilejoin)rZsubdir directorys r+r[z"index_directory..5s&NFBEEKK$$Y7Ns47)rxNzfExpected the lengths of `labels` to match the number of files in the target directory. len(labels) is z while we found z files in directory rint32r/zFound z files.z files belonging to z classes.r)sortedr r}r~listdirisdirr startswithendswithr6setrr_rArangermultiprocessingpool ThreadPool apply_asyncindex_subdirectorygetrzerosr print_msgcloserr RandomStater)rlabelsformats class_namesrr follow_linkssubdirsr class_indicesrresults filenamesdirpath labels_listrespartial_filenamespartial_labelsrDfname file_pathsrngs` r+index_directoryrsOP$ RUU[[00;< +Fuu{{  !1!1)V!DE((-s+!'NN6*  + !K;3w</ !!( )9+H [%K0@*ABCM    * * ,DGINgN    "-w?   K',/GGI)>>*&& ''' v;#i. (Dv;-/I/?@ ){!-  S[) 3y>+7;) %N2@F1q3~.. / ^$ $A %~VC N#37;<S^$%k"#9 .  JJLIIKBKL"%%++""9e4LJL <99$$S)Dii##D) Jii##D) F v{ **Ms 1Nc#,K|s*tjjj|}nt j||}t |dD]<\}}}t |D](}|j j|s#||f*>yw)N) followlinksc |dS)Nrxs r+z"iter_valid_files..ls QqTr-)key)r r}r~walkosrlowerr)rrrrroot_filesrs r+iter_valid_filesrgs~ uu{{ *wwyl; >:"aE] "E{{}%%g.Ek! ""s BB Bctjj|}t|||}g}g}|D]\}} |j ||t j jj|| } t j jj|tjj| |} |j | ||fS)aNRecursively walks directory and list image paths and their class index. Args: directory: string, target directory. class_indices: dict mapping class names to their index. follow_links: boolean, whether to recursively follow subdirectories (if False, we only list top-level images in `directory`). formats: Allowlist of file extensions to index (e.g. ".jpg", ".txt"). Returns: tuple `(filenames, labels)`. `filenames` is a list of relative file paths, and `labels` is a list of integer labels corresponding to these files. ) rpathbasenamerr6r r}r~rrelpath) rrrrdirname valid_filesrrrr absolute_path relative_paths r+rrrsggy)G"9lGDK FI"( e mG,- ((u5  (( RWW__]I>  ' ( f r-c(|s||fSt|t|z}|dk(r5tjdt||z d|d| }|d| }||fS|dk(r)tjd|d|| d}|| d}||fSt d|)aPotentially restict samples & labels to a training or validation split. Args: samples: List of elements. labels: List of corresponding labels. validation_split: Float, fraction of data to reserve for validation. subset: Subset of the data to return. Either "training", "validation", or None. If None, we return all of the data. Returns: tuple (samples, labels), potentially restricted to the specified subset. trainingzUsing z files for training.N validationz files for validation.z>`subset` must be either "training" or "validation", received: )rrrrr)samplesrvalidation_splitsubsetnum_val_sampless r+ get_training_or_validation_splitrs *S\9:O S\O344L M +O+,)/)* F? < VO#44JKL?*+,()* F?  **0 3  r-c&tjjj|}|dk(r.|j dtjj }|S|dk(r.|j fdtjj }|S)a]Create a tf.data.Dataset from the list/tuple of labels. Args: labels: list/tuple of labels to be converted into a tf.data.Dataset. label_mode: String describing the encoding of `labels`. Options are: - 'binary' indicates that the labels (there can be only 2) are encoded as `float32` scalars with values 0 or 1 (e.g. for `binary_crossentropy`). - 'categorical' means that the labels are mapped into a categorical vector. (e.g. for `categorical_crossentropy` loss). num_classes: number of classes of labels. Returns: A `Dataset` instance. binarycXtjtj|ddS)Nfloat32r{)axis)r expand_dimscastrs r+rz#labels_to_dataset..sbnnRWWQ %:Dr-)num_parallel_calls categoricalc0tj|SrW)r one_hot)r num_classess r+rz#labels_to_dataset..sbjjK0r-)r r r rmapr!)r label_moderlabel_dss ` r+labels_to_datasetrsww11&9HX<< D!ww//   O } $<< 0!ww//   Or-c|rd|cxkrdksntd||s|r|r|s td|dvrtd||r|r| tdyyy) aRaise errors in case of invalid argument values. Args: validation_split: float between 0 and 1, fraction of data to reserve for validation. subset: One of "training", "validation" or "both". Only used if `validation_split` is set. shuffle: Whether to shuffle the data. Either True or False. seed: random seed for shuffling and transformations. rrgz6`validation_split` must be between 0 and 1, received: zBIf `subset` is set, `validation_split` must be set, and inversely.)rrbothNzF`subset` must be either "training", "validation" or "both", received: NzIf using `validation_split` and shuffling the data, you must provide a `seed` argument, to make sure that there is no overlap between the training and validation subset.)r)rrrrs r+check_validation_split_argrs$4 8q 8 )* ,   F-=& P  == 117 :  G  B  )5Gr-)NNFN)TT)NTNF) __doc__multiprocessing.poolrrrr2rOnumpyrtensorflow.compat.v2compatv2r tf_keras.src.utilsr tensorflow.python.util.tf_exportrr,rr1r3rrrrrrrrrrrrr-r+rs- !!':)b1BFT#2T#t  ,^K\DN*!D  .   t+n":!H: r-