TinyBookyLLM (65M)

Trained on my laptop in about half an hour with a batch size of 4 (which only used about 5GB of memory, which is only about a fifth of what I have). Trained on about 32M tokens of gutenberg english. Just plain old Llama3 architecture, 16k BPE tokenizer, 640 hidden size, 2048 intermediate size, 8 q heads, 2 kv heads, RoPE, silu, 9 layers, sequence length 512, 16,000 steps, peak LR of 5e-5, cosine decay, no warmup.

Tests

Anyways, here's the tests: Prompt: The

The two years would be so much that they had not been a

great deal.


“I think I have no need of it,” she said, “that I am

not quite glad to say anything else.”
He was still very far from the most agreeable of his own; but he

was a man who could not help himself with the same words, and that he might

have been more than he did.  He knew how he meant, as he went to the

last, and he was ashamed of him, that he should not do anything for him,

he thought he would not get over. The thing was more important to him than

the other, and he did not wish to go into the world. He made himself

losely, and in a few days he had a good deal of his life, but this man

would never be a better thing than any one else, but that the people should

be allowed to give him a pleasure. But he must find the first time for

me—but he would be more than a woman to have, to whom he was doing

about a good thing, to make himself happy.


And then, as he came home, the man said: ‘You will see me

so, if you have come back to your face, and I shall be so sure that you

will be very good to me, and that I shall be too late to my mother, and that

you can’t tell me all about it, and that if I were a woman I may

know him, I shall take such an example as I love him.’


“Oh, don’t let me know that?” asked the old woman.


“Well, I suppose I didn’t care for her now, and I’m not

to-day. You’re only going to be sorry, and there are things like

what I have done, and you must be able to live on the way we shall be

like to you.”


She looked at him in silence, and tried to speak to her husband.


“But what is it? It’s the matter of being there for us, for

it’s nothing but that sort of thing. There’ll be something about

myself, and that makes me think of it. But I can’t want to keep some

Prompt: “My name is

“My name is a new, but you are very sorry. I am not going to be

done and have my work, and it has been the most beautiful of all the time that

I do not know what is called me. He knows the man who is in the way he

would be on the right side of his life, and he will not be so strong as by

the world.”


He was a good deal more than a certain thing; and that he had not the

fixedness of his own, when he was a great one. It made him an easy

interm which he had seen no longer, for he had never said any word. But he did

not get up, or could not even go into the house, nor make it a chance of

his mother's love.


In this case there was something but a single girl to say that she was

picked out of her mind with him, and that she would never have heard a little

about him.  That was what she wanted. She was getting away from him, and

he knew that he must be the only one! In the same moment the lady was

much more than ever, and he felt like the child of the poor, and he felt

so much more, and did not.


The day was in town a little boy’s face, and at last, having taken

him to take his money, he found himself in the presence of his own, and

in the heart of his voice he gave them, he had gone off to the country

and was to come out for some time, and they were too fond of him; then he

was afraid he should give them the truth.


But he was trying to know that he might be a master of the first to see him

that he was a very well-known man, and he thought he would be glad to tell him the

man that was, and, though, after all, he had just seen him, he would have

been the best of his wife, and he would not be able to see him.


When he had been sitting in the morning he told me how he was, and that he

could not help me, and that he was rather sure that he was in the least

better than he had said.


And now he looked about in a