Overview This code defines a FastAPI-based web server that facilitates: Fine-tuning and training of language models (e.g., LLMs like LLaMA) Dynamic tokenization optimizations (e.g., N-gram token enhancement) Dataset handling, chunking, and streaming Monitoring of system resources (RAM, CPU, GPU) Live training status, logs, and previewing training chunks A full web dashboard UI for controlling model training and chatting with the model Streaming chat endpoint with live responses File and Directory Initialization os.makedirs("./training_outputs", exist_ok=True) os.makedirs("./datasets", exist_ok=True) Ensures required directories (training_outputs, datasets) exist for storing models and training data. Web Server Initialization with FastAPI app = FastAPI() Uses FastAPI to define a RESTful and WebSocket-based server. Includes: CORS Middleware (allows access from any origin) Mounts static files for serving model outputs Core Functionalities Training Configuration & Status training_config = {...} training_status = {...} training_config: Default hyperparameters and paths training_status: Shared state tracking progress, ETA, logs, and previews Thread Safety training_lock = threading.Lock() Ensures that only one training session runs at a time. Helpers and Utilities System and File Utilities get_hardware_info() → Collects CPU/GPU availability get_system_status() → Retrieves memory and usage statistics get_new_output_dir() → Creates a timestamped output folder unload_torch_objects() → Frees GPU/CPU memory by deleting Torch objects N-gram Handling 1. Code Detection Heuristic is_code_ngram(text) Detects whether a text snippet is likely code, boosting its frequency during n-gram selection. 2. Frequent & Diverse N-Grams find_frequent_ngrams_diverse(...) Extracts frequent n-grams from tokenized data Prioritizes diversity Supports code boosting and similarity filtering 3. Tokenizer Optimization optimize_tokenizer_and_add_ngrams(...) Adds selected n-grams to tokenizer Sets average embedding vectors based on original token embeddings Dataset Iterators Provides several iterator functions to parse and stream training samples from CSV datasets: dialogplus_block_iter: Block-based samples for every assistant reply dialog_iter: Converts dialog trees into full chat examples column_iter: Simple line-by-line from a single column (for pretraining) stream_packed_chunk_generator: Converts data into fixed-size training chunks with optional overlap and padding Training Loop train_loop(config) This is the core training routine: Loads model/tokenizer Adds special tokens and optionally optimizes tokenizer Loads dataset in streaming fashion Supports: Full training LoRA fine-tuning Logs live statistics Saves model and cleans memory Web API Endpoints GET /hardware: Returns system hardware info GET /sysstatus: Returns CPU/GPU/RAM usage GET /trainings: Lists completed training runs POST /start: Starts a new training run POST /stop: Gracefully stops ongoing training GET /status: Returns current training state GET /logs: Returns recent logs GET /livepreview: Shows the current training sample WebSocket Endpoint @app.websocket("/ws/logs") Pushes real-time logs to connected clients. Model Inference (Chat) Supports interactive chatting with the trained model. Loading and Unloading: POST /load_inference POST /unload_inference Chat: POST /chat: Normal chat generation POST /chat_stream: Streaming chat (live token generation) Web Dashboard UI Served via the GET / endpoint. Features: Full control of training settings Resource monitoring (GPU/CPU) Live loss charts Log and preview window Chat interface with the model Dropdowns to select model/dataset The HTML and CSS build a modern green-themed "Matrix-style" interface. Special Notes Uses torch, transformers, datasets, and peft libraries GPU and MPS support Embedding average logic ensures newly added tokens are meaningful Supports templates like chat, dialogplus, text for flexible formats Modularity & Extensibility The code is highly modular. You can: Add new training strategies Extend the UI Switch from HuggingFace Transformers to other backends Add more analytics to the live dashboard