Description
DeciCoder 1B is a 1 billion parameter decoder-only code completion model trained on the Python, Java, and Javascript subsets of Starcoder Training Dataset.
Publishers
Deci AI Team
Submitted Version
August 15, 2023
Latest Version
N/A
Size
N/A
DeciCoder 1B is a 1 billion parameter decoder-only code completion model trained on the Python, Java, and Javascript subsets of Starcoder Training Dataset. The model uses Grouped Query Attention and has a context window of 2048 tokens. It was trained using a Fill-in-the-Middle training objective. The model’s architecture was generated by Deci’s proprietary Neural Architecture Search-based technology, AutoNAC.
| Parameters | Layers | Heads | Sequence Length | GQA num_key_value_heads | Hidden Size |
|---|---|---|---|---|---|
| 1.1B | 20 | 32 | 2048 | 4 | 2048 |
The model is intended to do single/multiline code completion from a context window of up to 2048k tokens. It is not an instruction model and commands like “Write a function that computes the absolute value of an integer,” won’t yield the desired results. A more effective approach is to frame instructions in the style of source code comments (e.g. # this function calculates the absolute value of an integer) or to present a function signature and docstring, enabling the model to complete the function’s body.
The model has undergone training with source code from Python, Java, and JavaScript. While the primary language in the source is English, it does contain other languages. Therefore, the model can produce code snippets given some context. However, there’s no assurance that the resulting code will function as expected. It might be suboptimal, contain bugs, or even exploits.
from transformers import AutoFeatureExtractor, AutoModelForImageClassification
extractor = AutoFeatureExtractor.from_pretrained("microsoft/resnet-50")
model = AutoModelForImageClassification.from_pretrained("microsoft/resnet-50")