Before training the model, I turned the strings to a numerical representation. To be more specific, after the text being split up into tokens, the preprocessing.StringLookup layer change each character into a numeric ID. Also, the purpose of this layer is to map strings from a vocabulary to integer indices. When creating this layer, I had vocabulary equal a list of the sorted set of the script from Shakespeare and mask_token equal 0. By setting mask_token to 0, no mask term will be added. In addition, this layer will convert form tokens to character IDs. After changing the layer to have invert=True, this layer will be able to recover the characters from the vectors of IDs and returns these characters in the form a tf.RaggedTensor. Due to invert equaling true, this layer will map indices to vocabulary items rather than mapping vocabulary items to indices. Also, a Ragged Tensor differs from other tensors by having one or more ragged dimensions or dimensions whose slices may have different lengths. In addition, I can use tf.strings.reduce_join to turn these characters back into strings. tf.strings.reduce_join is normally used for joining all strings into a single string or joining them along an axis. When using this function, I had the inputs equal the text split into tokens and the axis equal -1. For this model, the input will be a sequence of characters. These input sequences will have seq_length characters from the test. Additionally, I used the tf.data.Dataset.from_tensor_slices function to change the text vector into a stream of character indices. In addition, I used the batch method on the stream of character of indices to turn these individual characters into sequences of the desired size. For the batch method, I also used the element seq_lenth+1 and set the drop_remainder to True. Also, I used the function split_input_target(sequence) on the sequences of the desired size to make a dataset of (input, label) pairs in preparation for the training step. The purpose of this function is to take a sequence as input, duplicate, and shift a sequence to align the input and label for each timestep. In these pairs, input and label elements are sequences. For more clarification, the input represents the current character, and the label represents the next character. To prepare for inputting this data into the model, I shuffled the data and organizing it into batches. When doing this action, I used a batch size of 64 and a buffer size of 10000. This buffer size is used in shuffling the dataset. For the output of the following character at each time steps, I trained the model to predict this output.
The model built is part of a keras.Model subclass. This subclass is knowing for grouping layers into an object with training and inference features. The input layer of this model is tf.keras.layers.Embedding layer and is used in mapping each character-ID to a vector with embedding_dim dimensions. tf.keras.layers.Embedding layers are usually utilizing in turning positive integers (indexes) into dense vectors of fixed size. Another layer of this model is tf.keras.layers.GRU layer and a type of RNN that involves size units (RNN units). The output layer is tf.keras.layers.Dense layer and involves vocab_size outputs. These outputs involve logits for characters in the vocabulary and are the log-likelihood of each character according to the model. To clarify, this layer tf.keras.layers.Dense layer is a densely-connected NN layer. When building this model, I have to ensure that the vocabular size equals the ‘StringLookup’ layers. Also, embedding_dim should equal 256, and rnn_units equal 1024. To compile the model, I used the tf.keras.Model.compile method, the optimizer tf.keras.optimizers.Adam, and the standard tf.keras.losses.sparse_categorical_crossentropy loss function. The optimizer involves a stochastic gradient descent method that is based on adaptive estimation of first-order and second-order moments. In addition, this loss function is used in this model, because this function is utilized on the last dimension of the predictions. Due to the model returning logits, from_logits is set to True. Additionally, when training the model, I used the dataset of (input, label) pairs, 20 epochs, and tf.keras.callbacks.ModelCheckpoint. In this step, tf.keras.callbacks.ModelCheckpoint is meant to make sure that checkpoints are saved during training. Usually, tf.keras.callbacks.ModelCheckpoint is a callback meant to save the Keras model or model weights at some frequency.
One easy way to generate to use a loop. While using this look, I kept track of the model’s internal state. When calling the model, I give the model some text and an internal state. The model predicts the next character and its new state. This single step prediction is involved in preventing “[UNK]” from being generating, placing -inf at bad indexes, making the dense shape, match the length of the vocabulary, changing strings into token IDs, using the output logits in making token IDs, changing token IDs into characters, and helping continue generating text. By increasing the epochs in the training step, the model has a higher likelihood to form coherent sentences. To have the model generate text faster, the text generation should be batched. Overall, this generated text looks like Shakespeare writing.
tf.Tensor( [b”ROMEO:\nCome your whyer: then discover you hither hourly;\nevery time the princes fellow Dowbray’s stopp’d\nWith poison died in lawful thing,–which is my base;\nTell me of thispect old man’s rope-war!\n\nFirst Murderer:\nSo, now let me leave it, and we are you mark?\n\nBENVOLIO:\nTut, let me pardon for the sepil’s witch,\nHad not the great’st of it. Sirrah, here’s a life\nSemitch imairs; he was agody.\nThe best of them that would do them by this.\n\nWhither:\nFear believe him; staff if you lose nothing\nbut a signior of your blood we are old\nTwo frestrimght man: thou hast married ones.\n\nBUCKINGHAM:\nMarry, at the Tower is banish’d: amongst thee,\nHold, he hath stol’d, the fillion of your daughter too.\n\nHORTENSIO:\nAs all the trumpet, not a joy in sadness. Hata, say you\nFor mine honesty; marry, sir, inly my mother God!\nWhich God defired his noble accusation,\nIf she deny, Bianca, and thou diest;\nOur knees she bleeding, of all my friends.\nSigh’d troous that I am late me. Bast,\nNacrest this instantly, that host th” b”ROMEO:\nBut where is her own: she would speak, garted! Menough;\nI have no cause to sea again.\nTherefore, nurse! why, here comes the valiant voice o’ the\nrest; he should have made so fine a crutte\nBut it pleasers amongst me: but it concerns the\nshearing, fillinger, and present for the better\nThat crack’d with temporal of the minn.\nWhen you should tell what we are undoss those youth\nIs low with him, and I could grip not then in mine arms\nIn this way in the roof to them,\nAs by sad members. What has honour to forget thee?\nI have a dule, started from the land with ne’er\nIn storm people that e’er I will; for ‘tis desperate\nTo closon theirs. What news he is:\nHer chomsom and one unrest heat their country’s,\nLies me another lenath. But, gentle Northould!\n\nGLOUCESTER:\nYou were not kingly Bianca, by any of their love!\nO that defends not little love? I think, is well-may well:\nChirds appetty here have pardons itself if I\nwould I had them not and reft no man as shrour\nAnd fight with all my heart, and the “ b”ROMEO:\nGentlemen, go, see’ thy companion of the house.\n\nFirst Senator:\nCall him, fair cousin!\n\nProvost:\nHer hath kept very meet, past forth to me.\n\nPOMPEY:\nNo, but the king.\n\nGRUMIO:\nArt thou so much to morn umongst thee well!\nThen tread in secret care is but keep their abode.\nA name, are you say to my father named with him?\n\nPedant:\nKill’t please you forth, but now I show it not.\n\nDUKE VINCENTIO:\nYou are of good correct; thou shalt not others.\n\nROMEO:\nThen come there your myself become you.\n\nCORIOLANUS:\nHow comest thou forgive a story of my knowledge.\n\nFather:\nThough long consider’d, I must confess,\nGrace to my faith. Then I’ll be sacred vial,\nReceive it: have you conceive me for the woman of\nresidst, all speed to stand,\nPersuade the father, stay die time to make\nOnly my part from the muth of seater hurt. I long I hope\nTo give your haste most noble taster’s tortunes:\nThis pathing bady presents you, when it should dose,\nTo mistress mine own parts, that every day,\nTo chide away, my lady’s pra” b”ROMEO:\nKins they thought! although I be not know\nThis doubt me in my day shall serve, Forget, you\nwhere off that whips dead as hardongars,\nThe durins congemits for the father’s feath.\n\nNORTHUMBERLAND:\nNor I; for Clarence on, patricien, old York.\n\nYORK:\nThanks, well hath sure as strangery, but I am weeping,\nTo be thus to the fighty presence,\nOf these wealthy and not when thou livest,\nTo Burgun for instantly answering,\nAnd make your deeds contend with all my waged and\nThe fliers that remoster’d man’s life living,\nHold thine ear on the other force. The vasith\nDo much grenting philosove mens.\n\nNORFOLK:\nI am not in the skying conference\nwith thee of the head of the earth,–\nOf thy world gallentwou, tastes them away.\n\nKING HENRY VI:\nThere,or with him that slew the end of revenge\nThat your leader potench to afflicted with\nmen sickle, then hid punishment of mine\nOre she ever congemn’d with patience given, and marr’d him:\nThere lies I have a cousin’ use’t,\nAs being wench’s battle, that it presses, in” b”ROMEO:\nThere is the city is first tender fearful?\nWhere dragon will not other be done?\n\nPROSPERO:\nThere lie, thou lest a little.\n\nCORIOLANUS:\nHow! now! drunk; good Camillo?\n\nBIONDELLO:\nNay, no: the most strength is with them; and, was when I cannot do\nto utter always without sours.\n\nPOLIXENES:\nOf Marcius were the oath there no, myself Henry!\n\nJULIET:\nThen, like a cold simple grew you up with Gloucester’s death.\n\nBIANCA:\nThen bid me kill my hap away.\n\nHASTINGS:\nConsusion shines should litter wish me; I would prove\nA secret promise where they chosed by myself,\nIf they challenge granted neggers, brother,\nWas had they seek for Rome, he may provott.\nLasts after your garland. What, have I had find\nYour ladys and made the pooression of his attempt;\nAnd, brother, thus; he saints, return match.\nThen, drink, a horse; a lady’s chant,\nIt is a burthent with their child, and myself will;\nit will not vouch against him.\n\nFirst Murderer:\nShe is the queen to stretch atide;\nThis, he hath loig’d uncle Glouceste”], shape=(5,), dtype=string) ____________________________
Run time: 2.8644378185272217
ROMEO: I cannot get you go. This deed unspeet have alive to strike.
Nurse: Mark you; I’ll resolve you; I will serve, My bovel hot so.
Servant: Then might what says, it’s son, it is the fawners found And practist blood of blood; for earth, Olphort, Not like to put a time were there, which now take He being so heign of bliss’cian doth stand.
VIRGILIA: Let’s heard a man-catched by a lawful table. Stand was; Therefore take taster of this coat, but they are gone, ‘tis is some sovereign.
KING HENRY VI: Pardon me, Sicilia? For first the DesSin’s skitless till they call forgot; and to pry unconjudation Take the until to men we give the grieved motant. The valiant stuff, and water cannot entreat therefore.
First Gentleman: I would have none so sets the studies.
CLIFFORD: The lord’s feather and the sister of his age, When I am clean out. And thou hast son outhors, Your honour’d means at was fitting in my lands.
ROMEO: Ha, not a jottle better attempt most join, And proceed ear-hath play’d a fort
Run time: 2.131175994873047