Description
DeciCoder 6B is a 6 billion parameter decoder-only code completion model trained on the Python, Java, Javascript, Ruby, Rust, C++, C, and C# subsets of the Starcoder Training Dataset. The model uses variable Grouped Query Attention and has a context window of 4096 tokens.
Publishers
Deci AI Team
Submitted Version
January 17, 2024
Latest Version
N/A
Size
N/A
DeciCoder 6B is a 6 billion parameter decoder-only code completion model trained on the Python, Java, Javascript, Ruby, Rust, C++, C, and C# subsets of the Starcoder Training Dataset. The model uses variable Grouped Query Attention and has a context window of 4096 tokens. It was trained using a Fill-in-the-Middle training objective. The model’s architecture was generated by Deci’s proprietary Neural Architecture Search-based technology, AutoNAC and optimized to maximize performance on cost efficient hardware such as the Qualcomm Cloud AI 100.
Number of Parameters (In billions) | Layers | Heads | Sequence Length | GQA num_key_value_heads | |
DeciCoder 6B | 6B | 32 | 32 | 2K | Variable |
DeciCoder 6B employs variable Grouped Query Attention to refine the balance between operational efficiency and model quality. It consistently uses 32 queries/heads per layer and introduces variability in the GQA group parameter across different layers. Some layers mimic Multi-Query Attention with only a single group, while others incorporate multiple groups.
By customizing the grouping according to the specific needs of each layer, DeciCoder 6B achieves an ideal equilibrium between the speed of inference and the fidelity of its outputs. It effectively leverages the computational and memory efficiencies of grouped attention, while also benefiting from the detailed and varied attention patterns that this approach facilitates.
The model is intended to do single/multiline code completion from a context window of up to 4096k tokens. It is not an instruction model and commands like “Write a function that computes the absolute value of an integer,” won’t yield the desired results. A more effective approach is to frame instructions in the style of source code comments (e.g. # this function calculates the absolute value of an integer) or to present a function signature and docstring, enabling the model to complete the function’s body.
The model has undergone training with source code from Python, Java, JavaScript, Ruby, RUST, C++, C, and C#. While the primary language in the source is English, it does contain other languages. Therefore, the model can produce code snippets given some context. However, there’s no assurance that the resulting code will function as expected. It might be suboptimal, contain bugs, or even exploits.
The model can produce incorrect output. The model was trained on various data. While great efforts have been taken to clean the pretraining data, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.
Deci is ISO 27001
Certified
from transformers import AutoFeatureExtractor, AutoModelForImageClassification
extractor = AutoFeatureExtractor.from_pretrained("microsoft/resnet-50")
model = AutoModelForImageClassification.from_pretrained("microsoft/resnet-50")