USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference
-
Updated
Nov 15, 2024 - Python
#
ring-attention
Here are 4 public repositories matching this topic...
InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.
pytorch
multi-modal
gemma
pipeline-parallelism
transformers-models
tensor-parallelism
llava
llm-training
internlm
flash-attention
zero3
llm-framework
sequence-parallelism
internlm2
ring-attention
deepspeed-ulysses
llama3
910b
-
Updated
Nov 7, 2024 - Python
The official CLIP training codebase of Inf-CL: "Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss". A super memory-efficiency CLIP training scheme.
-
Updated
Oct 30, 2024 - Python
Packaged Ring Attention with Blockwise Transformers for Near-Infinite Context implemented in Jax + Flax.
-
Updated
May 10, 2024 - Python
Improve this page
Add a description, image, and links to the ring-attention topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the ring-attention topic, visit your repo's landing page and select "manage topics."