Pile202468

determining protease transformer steps (iterations):

To calculate the number of steps (iterations) needed to run for 4 epochs, we use the following formula:

$\text{num\_iterations}=\frac{\text{total\_tokens\_in\_dataset}\times \text{num\_epochs}}{\text{tokens\_per\_fwdbwd}}$

Given:

• total_tokens_in_dataset = 589222838
• num_epochs = 4
• tokens_per_fwdbwd = 262144 (obtained from default setting in modded nanogpt)

Plugging in the values:

$\text{num\_iterations}=\frac{589222838\times 4}{262144}\approx 8992$

So, we should run approximately 8992 steps (iterations) to achieve 4 epochs. We eventually choose to run 9k steps.