Skip to content

Commit

Permalink
Update README.md to be more detailed
Browse files Browse the repository at this point in the history
  • Loading branch information
jshuadvd committed Jul 16, 2024
1 parent 7a803d1 commit 7f64e2b
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ The LongRoPE model extends the context window of large language models beyond 2
angles = positions.unsqueeze(-1) * self.theta
sin_cos = torch.stack([angles.cos(), angles.sin()], dim=-1)
return sin_cos.view(*sin_cos.shape[:-2], -1)
```

2. Non-uniform Interpolation:

Expand All @@ -98,7 +99,7 @@ The LongRoPE model extends the context window of large language models beyond 2
interpolated_pos[..., 2 * i] *= scale
interpolated_pos[..., 2 * i + 1] *= scale
return interpolated_pos

```

### Progressive Extension Strategy

Expand Down

0 comments on commit 7f64e2b

Please sign in to comment.