title: Scaling Granite Code Models to 128K Context
summary: This paper introduces long-context Granite code models that support effective
  context windows of up to 128K tokens. Our solution for scaling context length
  of Granite 3B/8B code models from 2K/4K to 128K consists of a light-weight
  continual pretraining by gradually increasing its RoPE base frequency with
  repository-level file packing and length-upsampled long-context data.
  Additionally, we also release instruction-tuned models with long-context
  support which are derived by further finetuning the long context base models on
  a mix of permissively licensed short and long-context instruction-response
  pairs. While comparing to the original short-context Granite code models, our
  long-context models achieve significant improvements on long-context tasks
  without any noticeable performance degradation on regular code completion
  benchmarks (e.g., HumanEval). We release all our long-context Granite code
  models under an Apache 2.0 license for both research and commercial use.
paper_authors._id: 6699eca6c9111326dc2859ce
paper_authors.name: Matt Stallone
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:10:28.638Z
paper_authors._id: 63b5dfd50d5913eee487b1b7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b5dfd50d5913eee487b1b7/Av-QwMbfUScp0JpzoGGpK.jpeg
paper_authors.fullname: Matt
paper_authors.type: user
paper_authors.user: stallone
paper_authors._id: 6699eca6c9111326dc2859cf
paper_authors.name: Vaibhav Saxena
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:10:26.166Z
paper_authors._id: 64be2746ae436c88132a848f
paper_authors.avatarUrl: /avatars/cd8b08c310b459189b1008119a778ed8.svg
paper_authors.fullname: Vaibhav Saxena
paper_authors.type: user
paper_authors.user: vaisaxen
paper_authors._id: 6699eca6c9111326dc2859d0
paper_authors.name: Leonid Karlinsky
paper_authors._id: 6699eca6c9111326dc2859d1
paper_authors.name: Bridget McGinn
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-24T07:28:52.137Z
paper_authors._id: 641ddc37771e2336626dbb55
paper_authors.avatarUrl: /avatars/b71be4ef8a21d10746bdec6d271f055d.svg
paper_authors.fullname: Bridget M
paper_authors.type: user
paper_authors.user: brmcg
paper_authors._id: 6699eca6c9111326dc2859d2
paper_authors.name: Tim Bula
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-23T07:48:24.504Z
paper_authors._id: 6408a0e2b6a334f53e20d4ab
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6408a0e2b6a334f53e20d4ab/dIhkQsg-elWTAuqFRfYX-.jpeg
paper_authors.fullname: Tim Bula
paper_authors.type: user
paper_authors.user: timrbula
paper_authors._id: 6699eca6c9111326dc2859d3
paper_authors.name: Mayank Mishra
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:23:02.972Z
paper_authors._id: 62cd5057674cdb524450093d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cd5057674cdb524450093d/f67rlrdsKPRTLdXCXoa_X.jpeg
paper_authors.fullname: Mayank Mishra
paper_authors.type: user
paper_authors.user: mayank-mishra
paper_authors._id: 6699eca6c9111326dc2859d4
paper_authors.name: Adriana Meza Soria
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T07:25:39.046Z
paper_authors._id: 639bcaa2445b133a4e942436
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/639bcaa2445b133a4e942436/h4Oa1ljMQvKXpGCHhTJY5.jpeg
paper_authors.fullname: Adriana Meza Soria
paper_authors.type: user
paper_authors.user: amezasor
paper_authors._id: 6699eca6c9111326dc2859d5
paper_authors.name: Gaoyuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-28T08:01:09.758Z
paper_authors._id: 652431dc0e7792fa3aa2adc7
paper_authors.avatarUrl: /avatars/b865d99ff5267184b37ed11cd2bd5626.svg
paper_authors.fullname: Gaoyuan Zhang
paper_authors.type: user
paper_authors.user: gaoyuanz
paper_authors._id: 6699eca6c9111326dc2859d6
paper_authors.name: Aditya Prasad
paper_authors._id: 6699eca6c9111326dc2859d7
paper_authors.name: Yikang Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T07:44:34.309Z
paper_authors._id: 6463995939359568c63d740c
paper_authors.avatarUrl: /avatars/11caacefc68c1259dbc8764cc4340481.svg
paper_authors.fullname: Yikang Shen
paper_authors.type: user
paper_authors.user: YikangS
paper_authors._id: 6699eca6c9111326dc2859d8
paper_authors.name: Saptha Surendran
paper_authors._id: 6699eca6c9111326dc2859d9
paper_authors.name: Shanmukha Guttula
paper_authors._id: 6699eca6c9111326dc2859da
paper_authors.name: Hima Patel
paper_authors._id: 6699eca6c9111326dc2859db
paper_authors.name: Parameswaran Selvam
paper_authors._id: 6699eca6c9111326dc2859dc
paper_authors.name: Xuan-Hong Dang
paper_authors._id: 6699eca6c9111326dc2859dd
paper_authors.name: Yan Koyfman
paper_authors._id: 6699eca6c9111326dc2859de
paper_authors.name: Atin Sood
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-26T07:40:59.661Z
paper_authors._id: 635b494062fb2bc1b52d795d
paper_authors.avatarUrl: /avatars/26e2c80d3473670fa31342bccb75362a.svg
paper_authors.fullname: sood
paper_authors.type: user
paper_authors.user: atinsood
paper_authors._id: 6699eca6c9111326dc2859df
paper_authors.name: Rogerio Feris
paper_authors._id: 6699eca6c9111326dc2859e0
paper_authors.name: Nirmit Desai
paper_authors._id: 6699eca6c9111326dc2859e1
paper_authors.name: David D. Cox
paper_authors._id: 6699eca6c9111326dc2859e2
paper_authors.name: Ruchir Puri
paper_authors._id: 6699eca6c9111326dc2859e3
paper_authors.name: Rameswar Panda
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-07-19T04:33:43.669Z
paper_authors._id: 648a7542ac1fbb8a1a654645
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648a7542ac1fbb8a1a654645/jLPnY11EOuOa6ZHE7dVEX.jpeg
paper_authors.fullname: Rameswar Panda
paper_authors.type: user
paper_authors.user: rpand002
paper_title: Scaling Granite Code Models to 128K Context
paper_summary: This paper introduces long-context Granite code models that support effective
  context windows of up to 128K tokens. Our solution for scaling context length
  of Granite 3B/8B code models from 2K/4K to 128K consists of a light-weight
  continual pretraining by gradually increasing its RoPE base frequency with
  repository-level file packing and length-upsampled long-context data.
  Additionally, we also release instruction-tuned models with long-context
  support which are derived by further finetuning the long context base models on
  a mix of permissively licensed short and long-context instruction-response
  pairs. While comparing to the original short-context Granite code models, our
  long-context models achieve significant improvements on long-context tasks
  without any noticeable performance degradation on regular code completion
  benchmarks (e.g., HumanEval). We release all our long-context Granite code
  models under an Apache 2.0 license for both research and commercial use.
paper_ai_summary: Granite code models with extended context windows up to 128K tokens are achieved through lightweight continual pretraining and finetuning with increased RoPE frequencies and length-upsampled data, showing improvements in long-context tasks without degrading performance on standard benchmarks.
paper_ai_keywords: Granite code models RoPE repository-level file packing length-upsampled long-context data instruction-tuned models HumanEval long-context tasks code completion benchmarks