Request for Refund and Support: AutoTrain Space Failed, Unexpected Charges, and Billing Status Issue

Hello Hugging Face Support Team,
I am writing to request your assistance regarding several issues I encountered while using AutoTrain Spaces, which resulted in unexpected charges and a poor user experience.
Here are the main problems I faced:
AutoTrain Spaces Could Not Run New Models (Qwen3, Gemma, etc.):
Although the UI allowed me to select models like Qwen3 and Gemma, the backend environment did not support them due to outdated dependencies (transformers, accelerate, trl, etc.).
I repeatedly encountered errors such as “The checkpoint you are trying to load has model type qwen3 but Transformers does not recognize this architecture,” and “cannot import name 'TorchTensorParallelPlugin' from 'accelerate.utils'.”
I tried to fix this by editing the Dockerfile and upgrading dependencies, but the build process was extremely slow, and the environment was still not compatible.
Space Frequently Paused, Restarted, or Failed to Build:
The Space would often automatically pause or restart, especially after clicking resource status or starting training, causing all progress to be lost.
Even after following community advice (e.g., setting PAUSE_ON_FAILURE=0), the Space was unstable and unreliable.
Billing and Charges:
I was charged $2.78 for GPU usage, even though I was never able to successfully run any training job due to the above issues.
Additionally, my credit card was charged $10 (possibly as a pre-authorization or deposit), but I did not receive any value from the service.
After deleting the Space, I still see it listed as “Active” in the Billing section, which makes me concerned about further charges.
User Experience:
The AutoTrain UI suggests that users can easily train new models without technical intervention, but in reality, it requires advanced troubleshooting and manual environment fixes.
The time spent on repeated builds, failed jobs, and debugging was frustrating and did not match the advertised “no-code” experience.
My requests:
Please review my account and refund the $2.78 GPU charge, as well as clarify the $10 credit card charge (and refund if possible).
Please confirm that after deleting my Space, I will not incur any further charges, and explain why the Space still appears as “Active” in Billing.
I hope you can improve the AutoTrain product to better match the advertised experience, or at least provide clearer warnings about model/environment compatibility.
I can provide logs, screenshots, and my Space link if needed.
Thank you for your attention and support!
Best regards,
Betty

Hugging Face Username：Kaidomoto

![Image](https://github.com/user-attachments/assets/69a132e0-95e5-456e-a0c9-e95fc128eb04)

Below is the log after I turned off the PAUSE_ON_FAILURE:
关了。总算看到log了：===== Application Startup at 2025-06-07 05:36:50 =====


==========
== CUDA ==
==========

CUDA Version 12.1.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

Found existing installation: autotrain-advanced 0.8.37.dev0
Uninstalling autotrain-advanced-0.8.37.dev0:
  Successfully uninstalled autotrain-advanced-0.8.37.dev0
Collecting autotrain-advanced
  Downloading autotrain_advanced-0.8.36-py3-none-any.whl.metadata (21 kB)
Requirement already satisfied: albumentations==1.4.23 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.4.23)
Requirement already satisfied: datasets~=3.2.0 in ./env/lib/python3.10/site-packages (from datasets[vision]~=3.2.0->autotrain-advanced) (3.2.0)
Requirement already satisfied: evaluate==0.4.3 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.4.3)
Requirement already satisfied: ipadic==1.0.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.0.0)
Requirement already satisfied: jiwer==3.0.5 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (3.0.5)
Requirement already satisfied: joblib==1.4.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.4.2)
Requirement already satisfied: loguru==0.7.3 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.7.3)
Requirement already satisfied: pandas==2.2.3 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.2.3)
Requirement already satisfied: nltk==3.9.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (3.9.1)
Requirement already satisfied: optuna==4.1.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (4.1.0)
Requirement already satisfied: Pillow==11.0.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (11.0.0)
Requirement already satisfied: sacremoses==0.1.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.1.1)
Requirement already satisfied: scikit-learn==1.6.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.6.0)
Requirement already satisfied: sentencepiece==0.2.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.2.0)
Requirement already satisfied: tqdm==4.67.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (4.67.1)
Requirement already satisfied: werkzeug==3.1.3 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (3.1.3)
Requirement already satisfied: xgboost==2.1.3 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.1.3)
Requirement already satisfied: huggingface-hub==0.27.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.27.0)
Requirement already satisfied: requests==2.32.3 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.32.3)
Requirement already satisfied: einops==0.8.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.8.0)
Requirement already satisfied: packaging==24.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (24.2)
Requirement already satisfied: cryptography==44.0.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (44.0.0)
Requirement already satisfied: nvitop==1.3.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.3.2)
Requirement already satisfied: tensorboard==2.18.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.18.0)
Requirement already satisfied: peft==0.14.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.14.0)
Requirement already satisfied: trl==0.13.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.13.0)
Requirement already satisfied: tiktoken==0.8.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.8.0)
Requirement already satisfied: transformers==4.48.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (4.48.0)
Requirement already satisfied: accelerate==1.2.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.2.1)
Requirement already satisfied: rouge-score==0.1.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.1.2)
Requirement already satisfied: py7zr==0.22.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.22.0)
Requirement already satisfied: fastapi==0.115.6 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.115.6)
Requirement already satisfied: uvicorn==0.34.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.34.0)
Requirement already satisfied: python-multipart==0.0.20 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.0.20)
Requirement already satisfied: pydantic==2.10.4 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.10.4)
Requirement already satisfied: hf-transfer in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.1.9)
Requirement already satisfied: pyngrok==7.2.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (7.2.1)
Requirement already satisfied: authlib==1.4.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.4.0)
Requirement already satisfied: itsdangerous==2.2.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.2.0)
Requirement already satisfied: seqeval==1.2.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.2.2)
Requirement already satisfied: httpx==0.28.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.28.1)
Requirement already satisfied: pyyaml==6.0.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (6.0.2)
Requirement already satisfied: timm==1.0.12 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.0.12)
Requirement already satisfied: torchmetrics==1.6.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.6.0)
Requirement already satisfied: pycocotools==2.0.8 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.0.8)
Requirement already satisfied: sentence-transformers==3.3.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (3.3.1)
Requirement already satisfied: bitsandbytes==0.45.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.45.0)
Requirement already satisfied: numpy<3.0.0,>=1.17 in ./env/lib/python3.10/site-packages (from accelerate==1.2.1->autotrain-advanced) (2.0.1)
Requirement already satisfied: psutil in ./env/lib/python3.10/site-packages (from accelerate==1.2.1->autotrain-advanced) (6.1.1)
Requirement already satisfied: torch>=1.10.0 in ./env/lib/python3.10/site-packages (from accelerate==1.2.1->autotrain-advanced) (2.4.0)
Requirement already satisfied: safetensors>=0.4.3 in ./env/lib/python3.10/site-packages (from accelerate==1.2.1->autotrain-advanced) (0.5.2)
Requirement already satisfied: scipy>=1.10.0 in ./env/lib/python3.10/site-packages (from albumentations==1.4.23->autotrain-advanced) (1.15.1)
Requirement already satisfied: albucore==0.0.21 in ./env/lib/python3.10/site-packages (from albumentations==1.4.23->autotrain-advanced) (0.0.21)
Requirement already satisfied: eval-type-backport in ./env/lib/python3.10/site-packages (from albumentations==1.4.23->autotrain-advanced) (0.2.2)
Requirement already satisfied: opencv-python-headless>=4.9.0.80 in ./env/lib/python3.10/site-packages (from albumentations==1.4.23->autotrain-advanced) (4.11.0.86)
Requirement already satisfied: typing_extensions>=4.8.0 in ./env/lib/python3.10/site-packages (from bitsandbytes==0.45.0->autotrain-advanced) (4.12.2)
Requirement already satisfied: cffi>=1.12 in ./env/lib/python3.10/site-packages (from cryptography==44.0.0->autotrain-advanced) (1.17.1)
Requirement already satisfied: dill in ./env/lib/python3.10/site-packages (from evaluate==0.4.3->autotrain-advanced) (0.3.8)
Requirement already satisfied: xxhash in ./env/lib/python3.10/site-packages (from evaluate==0.4.3->autotrain-advanced) (3.5.0)
Requirement already satisfied: multiprocess in ./env/lib/python3.10/site-packages (from evaluate==0.4.3->autotrain-advanced) (0.70.16)
Requirement already satisfied: fsspec>=2021.05.0 in ./env/lib/python3.10/site-packages (from fsspec[http]>=2021.05.0->evaluate==0.4.3->autotrain-advanced) (2024.9.0)
Requirement already satisfied: starlette<0.42.0,>=0.40.0 in ./env/lib/python3.10/site-packages (from fastapi==0.115.6->autotrain-advanced) (0.41.3)
Requirement already satisfied: anyio in ./env/lib/python3.10/site-packages (from httpx==0.28.1->autotrain-advanced) (4.8.0)
Requirement already satisfied: certifi in ./env/lib/python3.10/site-packages (from httpx==0.28.1->autotrain-advanced) (2024.12.14)
Requirement already satisfied: httpcore==1.* in ./env/lib/python3.10/site-packages (from httpx==0.28.1->autotrain-advanced) (1.0.7)
Requirement already satisfied: idna in ./env/lib/python3.10/site-packages (from httpx==0.28.1->autotrain-advanced) (3.7)
Requirement already satisfied: filelock in ./env/lib/python3.10/site-packages (from huggingface-hub==0.27.0->autotrain-advanced) (3.13.1)
Requirement already satisfied: click<9.0.0,>=8.1.3 in ./env/lib/python3.10/site-packages (from jiwer==3.0.5->autotrain-advanced) (8.1.8)
Requirement already satisfied: rapidfuzz<4,>=3 in ./env/lib/python3.10/site-packages (from jiwer==3.0.5->autotrain-advanced) (3.11.0)
Requirement already satisfied: regex>=2021.8.3 in ./env/lib/python3.10/site-packages (from nltk==3.9.1->autotrain-advanced) (2024.11.6)
Requirement already satisfied: nvidia-ml-py<12.536.0a0,>=11.450.51 in ./env/lib/python3.10/site-packages (from nvitop==1.3.2->autotrain-advanced) (12.535.161)
Requirement already satisfied: cachetools>=1.0.1 in ./env/lib/python3.10/site-packages (from nvitop==1.3.2->autotrain-advanced) (5.5.0)
Requirement already satisfied: termcolor>=1.0.0 in ./env/lib/python3.10/site-packages (from nvitop==1.3.2->autotrain-advanced) (2.5.0)
Requirement already satisfied: alembic>=1.5.0 in ./env/lib/python3.10/site-packages (from optuna==4.1.0->autotrain-advanced) (1.14.1)
Requirement already satisfied: colorlog in ./env/lib/python3.10/site-packages (from optuna==4.1.0->autotrain-advanced) (6.9.0)
Requirement already satisfied: sqlalchemy>=1.4.2 in ./env/lib/python3.10/site-packages (from optuna==4.1.0->autotrain-advanced) (2.0.37)
Requirement already satisfied: python-dateutil>=2.8.2 in ./env/lib/python3.10/site-packages (from pandas==2.2.3->autotrain-advanced) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in ./env/lib/python3.10/site-packages (from pandas==2.2.3->autotrain-advanced) (2024.2)
Requirement already satisfied: tzdata>=2022.7 in ./env/lib/python3.10/site-packages (from pandas==2.2.3->autotrain-advanced) (2024.2)
Requirement already satisfied: texttable in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (1.7.0)
Requirement already satisfied: pycryptodomex>=3.16.0 in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (3.21.0)
Requirement already satisfied: pyzstd>=0.15.9 in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (0.16.2)
Requirement already satisfied: pyppmd<1.2.0,>=1.1.0 in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (1.1.1)
Requirement already satisfied: pybcj<1.1.0,>=1.0.0 in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (1.0.3)
Requirement already satisfied: multivolumefile>=0.2.3 in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (0.2.3)
Requirement already satisfied: inflate64<1.1.0,>=1.0.0 in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (1.0.1)
Requirement already satisfied: brotli>=1.1.0 in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (1.1.0)
Requirement already satisfied: matplotlib>=2.1.0 in ./env/lib/python3.10/site-packages (from pycocotools==2.0.8->autotrain-advanced) (3.10.0)
Requirement already satisfied: annotated-types>=0.6.0 in ./env/lib/python3.10/site-packages (from pydantic==2.10.4->autotrain-advanced) (0.7.0)
Requirement already satisfied: pydantic-core==2.27.2 in ./env/lib/python3.10/site-packages (from pydantic==2.10.4->autotrain-advanced) (2.27.2)
Requirement already satisfied: charset-normalizer<4,>=2 in ./env/lib/python3.10/site-packages (from requests==2.32.3->autotrain-advanced) (3.3.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./env/lib/python3.10/site-packages (from requests==2.32.3->autotrain-advanced) (2.3.0)
Requirement already satisfied: absl-py in ./env/lib/python3.10/site-packages (from rouge-score==0.1.2->autotrain-advanced) (2.1.0)
Requirement already satisfied: six>=1.14.0 in ./env/lib/python3.10/site-packages (from rouge-score==0.1.2->autotrain-advanced) (1.17.0)
Requirement already satisfied: threadpoolctl>=3.1.0 in ./env/lib/python3.10/site-packages (from scikit-learn==1.6.0->autotrain-advanced) (3.5.0)
Requirement already satisfied: grpcio>=1.48.2 in ./env/lib/python3.10/site-packages (from tensorboard==2.18.0->autotrain-advanced) (1.69.0)
Requirement already satisfied: markdown>=2.6.8 in ./env/lib/python3.10/site-packages (from tensorboard==2.18.0->autotrain-advanced) (3.7)
Requirement already satisfied: protobuf!=4.24.0,>=3.19.6 in ./env/lib/python3.10/site-packages (from tensorboard==2.18.0->autotrain-advanced) (5.29.3)
Requirement already satisfied: setuptools>=41.0.0 in ./env/lib/python3.10/site-packages (from tensorboard==2.18.0->autotrain-advanced) (75.1.0)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in ./env/lib/python3.10/site-packages (from tensorboard==2.18.0->autotrain-advanced) (0.7.2)
Requirement already satisfied: torchvision in ./env/lib/python3.10/site-packages (from timm==1.0.12->autotrain-advanced) (0.19.0)
Requirement already satisfied: lightning-utilities>=0.8.0 in ./env/lib/python3.10/site-packages (from torchmetrics==1.6.0->autotrain-advanced) (0.11.9)
Requirement already satisfied: tokenizers<0.22,>=0.21 in ./env/lib/python3.10/site-packages (from transformers==4.48.0->autotrain-advanced) (0.21.0)
Requirement already satisfied: rich in ./env/lib/python3.10/site-packages (from trl==0.13.0->autotrain-advanced) (13.9.4)
Requirement already satisfied: h11>=0.8 in ./env/lib/python3.10/site-packages (from uvicorn==0.34.0->autotrain-advanced) (0.14.0)
Requirement already satisfied: MarkupSafe>=2.1.1 in ./env/lib/python3.10/site-packages (from werkzeug==3.1.3->autotrain-advanced) (2.1.3)
Requirement already satisfied: nvidia-nccl-cu12 in ./env/lib/python3.10/site-packages (from xgboost==2.1.3->autotrain-advanced) (2.24.3)
Requirement already satisfied: stringzilla>=3.10.4 in ./env/lib/python3.10/site-packages (from albucore==0.0.21->albumentations==1.4.23->autotrain-advanced) (3.11.3)
Requirement already satisfied: simsimd>=5.9.2 in ./env/lib/python3.10/site-packages (from albucore==0.0.21->albumentations==1.4.23->autotrain-advanced) (6.2.1)
Requirement already satisfied: pyarrow>=15.0.0 in ./env/lib/python3.10/site-packages (from datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (19.0.0)
Requirement already satisfied: aiohttp in ./env/lib/python3.10/site-packages (from datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (3.11.11)
Requirement already satisfied: Mako in ./env/lib/python3.10/site-packages (from alembic>=1.5.0->optuna==4.1.0->autotrain-advanced) (1.3.8)
Requirement already satisfied: pycparser in ./env/lib/python3.10/site-packages (from cffi>=1.12->cryptography==44.0.0->autotrain-advanced) (2.22)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (2.4.4)
Requirement already satisfied: aiosignal>=1.1.2 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (1.3.2)
Requirement already satisfied: async-timeout<6.0,>=4.0 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (5.0.1)
Requirement already satisfied: attrs>=17.3.0 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (24.3.0)
Requirement already satisfied: frozenlist>=1.1.1 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (1.5.0)
Requirement already satisfied: multidict<7.0,>=4.5 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (6.1.0)
Requirement already satisfied: propcache>=0.2.0 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (0.2.1)
Requirement already satisfied: yarl<2.0,>=1.17.0 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (1.18.3)
Requirement already satisfied: contourpy>=1.0.1 in ./env/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools==2.0.8->autotrain-advanced) (1.3.1)
Requirement already satisfied: cycler>=0.10 in ./env/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools==2.0.8->autotrain-advanced) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in ./env/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools==2.0.8->autotrain-advanced) (4.55.3)
Requirement already satisfied: kiwisolver>=1.3.1 in ./env/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools==2.0.8->autotrain-advanced) (1.4.8)
Requirement already satisfied: pyparsing>=2.3.1 in ./env/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools==2.0.8->autotrain-advanced) (3.2.1)
Requirement already satisfied: greenlet!=0.4.17 in ./env/lib/python3.10/site-packages (from sqlalchemy>=1.4.2->optuna==4.1.0->autotrain-advanced) (3.1.1)
Requirement already satisfied: exceptiongroup>=1.0.2 in ./env/lib/python3.10/site-packages (from anyio->httpx==0.28.1->autotrain-advanced) (1.2.2)
Requirement already satisfied: sniffio>=1.1 in ./env/lib/python3.10/site-packages (from anyio->httpx==0.28.1->autotrain-advanced) (1.3.1)
Requirement already satisfied: sympy in ./env/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==1.2.1->autotrain-advanced) (1.13.3)
Requirement already satisfied: networkx in ./env/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==1.2.1->autotrain-advanced) (3.4.2)
Requirement already satisfied: jinja2 in ./env/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==1.2.1->autotrain-advanced) (3.1.4)
Requirement already satisfied: markdown-it-py>=2.2.0 in ./env/lib/python3.10/site-packages (from rich->trl==0.13.0->autotrain-advanced) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in ./env/lib/python3.10/site-packages (from rich->trl==0.13.0->autotrain-advanced) (2.19.1)
Requirement already satisfied: mdurl~=0.1 in ./env/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich->trl==0.13.0->autotrain-advanced) (0.1.2)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in ./env/lib/python3.10/site-packages (from sympy->torch>=1.10.0->accelerate==1.2.1->autotrain-advanced) (1.3.0)
Downloading autotrain_advanced-0.8.36-py3-none-any.whl (341 kB)
Installing collected packages: autotrain-advanced
Successfully installed autotrain-advanced-0.8.36
INFO     | 2025-06-07 05:37:06 | autotrain.app.ui_routes:<module>:31 - Starting AutoTrain...
INFO     | 2025-06-07 05:37:08 | autotrain.app.ui_routes:<module>:315 - AutoTrain started successfully
INFO     | 2025-06-07 05:37:08 | autotrain.app.app:<module>:13 - Starting AutoTrain...
INFO     | 2025-06-07 05:37:08 | autotrain.app.app:<module>:23 - AutoTrain version: 0.8.36
INFO     | 2025-06-07 05:37:08 | autotrain.app.app:<module>:24 - AutoTrain started successfully
INFO:     Started server process [93]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:7860 (Press CTRL+C to quit)
INFO:     10.16.10.84:50276 - "GET /?logs=container&__sign=eyJhbGciOiJFZERTQSJ9.eyJyZWFkIjp0cnVlLCJwZXJtaXNzaW9ucyI6eyJyZXBvLmNvbnRlbnQucmVhZCI6dHJ1ZX0sIm9uQmVoYWxmT2YiOnsia2luZCI6InVzZXIiLCJfaWQiOiI2N2RjZWRlNTQwMDRiMmYwZGVmYWRiN2EiLCJ1c2VyIjoia2FpZG9tb3RvIiwic2Vzc2lvbklkIjoiNjgzYzQ3NzIxNzEzODZhMGY5MDljZmE4In0sImlhdCI6MTc0OTI3NDYzMSwic3ViIjoiL3NwYWNlcy9rYWlkb21vdG8vcXdlbjNfOGJfbG9yYV9kZW1vIiwiZXhwIjoxNzQ5MzYxMDMxLCJpc3MiOiJodHRwczovL2h1Z2dpbmdmYWNlLmNvIn0.EB8wdK3xBE106u2VGnqMPkT1kJfvnNJg21mXRwK49OuvDw5pBzvg7NdmgibDZU5NmW39upp57j9_tL7RYKo6CQ HTTP/1.1" 307 Temporary Redirect
INFO:     10.16.10.84:50276 - "GET /?logs=container&__sign=eyJhbGciOiJFZERTQSJ9.eyJyZWFkIjp0cnVlLCJwZXJtaXNzaW9ucyI6eyJyZXBvLmNvbnRlbnQucmVhZCI6dHJ1ZX0sIm9uQmVoYWxmT2YiOnsia2luZCI6InVzZXIiLCJfaWQiOiI2N2RjZWRlNTQwMDRiMmYwZGVmYWRiN2EiLCJ1c2VyIjoia2FpZG9tb3RvIiwic2Vzc2lvbklkIjoiNjgzYzQ3NzIxNzEzODZhMGY5MDljZmE4In0sImlhdCI6MTc0OTI3NDYzMSwic3ViIjoiL3NwYWNlcy9rYWlkb21vdG8vcXdlbjNfOGJfbG9yYV9kZW1vIiwiZXhwIjoxNzQ5MzYxMDMxLCJpc3MiOiJodHRwczovL2h1Z2dpbmdmYWNlLmNvIn0.EB8wdK3xBE106u2VGnqMPkT1kJfvnNJg21mXRwK49OuvDw5pBzvg7NdmgibDZU5NmW39upp57j9_tL7RYKo6CQ HTTP/1.1" 307 Temporary Redirect
INFO:     10.16.39.72:38797 - "GET /ui/?logs=container&__sign=eyJhbGciOiJFZERTQSJ9.eyJyZWFkIjp0cnVlLCJwZXJtaXNzaW9ucyI6eyJyZXBvLmNvbnRlbnQucmVhZCI6dHJ1ZX0sIm9uQmVoYWxmT2YiOnsia2luZCI6InVzZXIiLCJfaWQiOiI2N2RjZWRlNTQwMDRiMmYwZGVmYWRiN2EiLCJ1c2VyIjoia2FpZG9tb3RvIiwic2Vzc2lvbklkIjoiNjgzYzQ3NzIxNzEzODZhMGY5MDljZmE4In0sImlhdCI6MTc0OTI3NDYzMSwic3ViIjoiL3NwYWNlcy9rYWlkb21vdG8vcXdlbjNfOGJfbG9yYV9kZW1vIiwiZXhwIjoxNzQ5MzYxMDMxLCJpc3MiOiJodHRwczovL2h1Z2dpbmdmYWNlLmNvIn0.EB8wdK3xBE106u2VGnqMPkT1kJfvnNJg21mXRwK49OuvDw5pBzvg7NdmgibDZU5NmW39upp57j9_tL7RYKo6CQ HTTP/1.1" 200 OK
INFO:     10.16.10.84:50276 - "GET /static/scripts/poll.js?cb=2025-06-07%2005:37:11 HTTP/1.1" 200 OK
INFO:     10.16.39.72:38797 - "GET /static/scripts/listeners.js?cb=2025-06-07%2005:37:11 HTTP/1.1" 200 OK
INFO:     10.16.13.110:63705 - "GET /static/scripts/fetch_data_and_update_models.js?cb=2025-06-07%2005:37:11 HTTP/1.1" 200 OK
INFO:     10.16.10.84:3123 - "GET /static/scripts/utils.js?cb=2025-06-07%2005:37:11 HTTP/1.1" 200 OK
INFO:     10.16.10.84:50276 - "GET /static/scripts/logs.js?cb=2025-06-07%2005:37:11 HTTP/1.1" 200 OK
INFO     | 2025-06-07 05:37:12 | autotrain.app.ui_routes:fetch_params:415 - Task: llm:sft
INFO:     10.16.13.110:63705 - "GET /ui/params/llm%3Asft/basic HTTP/1.1" 200 OK
INFO:     10.16.39.153:62767 - "GET /ui/accelerators HTTP/1.1" 200 OK
INFO:     10.16.10.84:50276 - "GET /ui/model_choices/llm%3Asft HTTP/1.1" 200 OK
INFO:     10.16.39.72:38797 - "GET /ui/is_model_training HTTP/1.1" 200 OK
INFO:     10.16.39.72:24830 - "GET /ui/is_model_training HTTP/1.1" 200 OK
INFO:     10.16.13.110:29953 - "GET /ui/is_model_training HTTP/1.1" 200 OK
INFO:     10.16.39.153:58623 - "GET /ui/accelerators HTTP/1.1" 200 OK
INFO:     10.16.13.110:27165 - "GET /ui/is_model_training HTTP/1.1" 200 OK
INFO:     10.16.10.84:21015 - "GET /ui/accelerators HTTP/1.1" 200 OK
INFO:     10.16.10.84:58821 - "GET /ui/is_model_training HTTP/1.1" 200 OK
INFO:     10.16.10.84:39561 - "GET /ui/is_model_training HTTP/1.1" 200 OK
INFO     | 2025-06-07 05:37:39 | autotrain.app.ui_routes:handle_form:540 - hardware: local-ui
INFO     | 2025-06-07 05:37:39 | autotrain.app.ui_routes:handle_form:656 - Task: lm_training
INFO     | 2025-06-07 05:37:39 | autotrain.app.ui_routes:handle_form:657 - Column mapping: {'text': 'text'}

Saving the dataset (0/1 shards):   0%|          | 0/500 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 500/500 [00:00<00:00, 203153.35 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 500/500 [00:00<00:00, 193732.29 examples/s]

Saving the dataset (0/1 shards):   0%|          | 0/500 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 500/500 [00:00<00:00, 266915.11 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 500/500 [00:00<00:00, 252790.74 examples/s]
INFO     | 2025-06-07 05:37:39 | autotrain.backends.local:create:20 - Starting local training...
INFO     | 2025-06-07 05:37:39 | autotrain.commands:launch_command:514 - ['accelerate', 'launch', '--num_machines', '1', '--num_processes', '1', '--mixed_precision', 'fp16', '-m', 'autotrain.trainers.clm', '--training_config', 'autotrain-g7k97-lotkf/training_params.json']
INFO     | 2025-06-07 05:37:39 | autotrain.commands:launch_command:515 - {'model': 'Qwen/Qwen3-4B', 'project_name': 'autotrain-g7k97-lotkf', 'data_path': 'autotrain-g7k97-lotkf/autotrain-data', 'train_split': 'train', 'valid_split': None, 'add_eos_token': True, 'block_size': 1024, 'model_max_length': 1024, 'padding': 'right', 'trainer': 'sft', 'use_flash_attention_2': False, 'log': 'tensorboard', 'disable_gradient_checkpointing': False, 'logging_steps': -1, 'eval_strategy': 'epoch', 'save_total_limit': 1, 'auto_find_batch_size': False, 'mixed_precision': 'fp16', 'lr': 3e-05, 'epochs': 3, 'batch_size': 2, 'warmup_ratio': 0.1, 'gradient_accumulation': 4, 'optimizer': 'adamw_torch', 'scheduler': 'linear', 'weight_decay': 0.0, 'max_grad_norm': 1.0, 'seed': 42, 'chat_template': 'none', 'quantization': 'int4', 'target_modules': 'all-linear', 'merge_adapter': False, 'peft': True, 'lora_r': 16, 'lora_alpha': 32, 'lora_dropout': 0.05, 'model_ref': None, 'dpo_beta': 0.1, 'max_prompt_length': 128, 'max_completion_length': None, 'prompt_text_column': 'autotrain_prompt', 'text_column': 'autotrain_text', 'rejected_text_column': 'autotrain_rejected_text', 'push_to_hub': True, 'username': 'kaidomoto', 'token': '*****', 'unsloth': False, 'distributed_backend': 'ddp'}
INFO     | 2025-06-07 05:37:39 | autotrain.backends.local:create:25 - Training PID: 152
INFO:     10.16.10.84:39561 - "POST /ui/create_project HTTP/1.1" 200 OK
INFO:     10.16.13.110:44549 - "GET /ui/is_model_training HTTP/1.1" 200 OK
INFO:     10.16.10.84:21609 - "GET /ui/accelerators HTTP/1.1" 200 OK
The following values were not passed to `accelerate launch` and had defaults used instead:
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
INFO     | 2025-06-07 05:37:47 | autotrain.trainers.clm.train_clm_sft:train:11 - Starting SFT training...
INFO     | 2025-06-07 05:37:47 | autotrain.trainers.clm.utils:process_input_data:487 - loading dataset from disk
INFO     | 2025-06-07 05:37:47 | autotrain.trainers.clm.utils:process_input_data:550 - Train data: Dataset({
    features: ['autotrain_text'],
    num_rows: 500
})
INFO     | 2025-06-07 05:37:47 | autotrain.trainers.clm.utils:process_input_data:551 - Valid data: None
INFO:     10.16.39.72:55711 - "GET /ui/is_model_training HTTP/1.1" 200 OK
INFO     | 2025-06-07 05:37:48 | autotrain.trainers.clm.utils:configure_logging_steps:671 - configuring logging steps
INFO     | 2025-06-07 05:37:48 | autotrain.trainers.clm.utils:configure_logging_steps:684 - Logging steps: 25
INFO     | 2025-06-07 05:37:48 | autotrain.trainers.clm.utils:configure_training_args:723 - configuring training args
INFO     | 2025-06-07 05:37:48 | autotrain.trainers.clm.utils:configure_block_size:801 - Using block size 1024
ERROR    | 2025-06-07 05:37:48 | autotrain.trainers.common:wrapper:215 - train has failed due to an exception: Traceback (most recent call last):
  File "/app/env/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1071, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/app/env/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 773, in __getitem__
    raise KeyError(key)
KeyError: 'qwen3'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/env/lib/python3.10/site-packages/autotrain/trainers/common.py", line 212, in wrapper
    return func(*args, **kwargs)
  File "/app/env/lib/python3.10/site-packages/autotrain/trainers/clm/__main__.py", line 28, in train
    train_sft(config)
  File "/app/env/lib/python3.10/site-packages/autotrain/trainers/clm/train_clm_sft.py", line 27, in train
    model = utils.get_model(config, tokenizer)
  File "/app/env/lib/python3.10/site-packages/autotrain/trainers/clm/utils.py", line 857, in get_model
    model_config = AutoConfig.from_pretrained(
  File "/app/env/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1073, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `qwen3` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git`

ERROR    | 2025-06-07 05:37:48 | autotrain.trainers.common:wrapper:216 - The checkpoint you are trying to load has model type `qwen3` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git`
INFO     | 2025-06-07 05:37:52 | autotrain.app.utils:get_running_jobs:40 - Killing PID: 152
INFO     | 2025-06-07 05:37:52 | autotrain.app.utils:kill_process_by_pid:90 - Sent SIGTERM to process with PID 152
INFO:     10.16.10.84:16372 - "GET /ui/is_model_training HTTP/1.1" 200 OK


===========================================================
AND BELOW IS THE LOG AFTER I REVISED DOCKER FILE into:
FROM huggingface/autotrain-advanced:latest
RUN pip install --upgrade pip && pip install git+https://github.com/huggingface/transformers.git
CMD pip uninstall -y autotrain-advanced && pip install -U autotrain-advanced && autotrain app --host 0.0.0.0 --port 7860 --workers 1
-----------LOG FROM HERE--------------------------------------------


CUDA Version 12.1.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

Found existing installation: autotrain-advanced 0.8.37.dev0
Uninstalling autotrain-advanced-0.8.37.dev0:
  Successfully uninstalled autotrain-advanced-0.8.37.dev0
Collecting autotrain-advanced
  Downloading autotrain_advanced-0.8.36-py3-none-any.whl.metadata (21 kB)
Requirement already satisfied: albumentations==1.4.23 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.4.23)
Requirement already satisfied: datasets~=3.2.0 in ./env/lib/python3.10/site-packages (from datasets[vision]~=3.2.0->autotrain-advanced) (3.2.0)
Requirement already satisfied: evaluate==0.4.3 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.4.3)
Requirement already satisfied: ipadic==1.0.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.0.0)
Requirement already satisfied: jiwer==3.0.5 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (3.0.5)
Requirement already satisfied: joblib==1.4.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.4.2)
Requirement already satisfied: loguru==0.7.3 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.7.3)
Requirement already satisfied: pandas==2.2.3 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.2.3)
Requirement already satisfied: nltk==3.9.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (3.9.1)
Requirement already satisfied: optuna==4.1.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (4.1.0)
Requirement already satisfied: Pillow==11.0.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (11.0.0)
Requirement already satisfied: sacremoses==0.1.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.1.1)
Requirement already satisfied: scikit-learn==1.6.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.6.0)
Requirement already satisfied: sentencepiece==0.2.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.2.0)
Requirement already satisfied: tqdm==4.67.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (4.67.1)
Requirement already satisfied: werkzeug==3.1.3 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (3.1.3)
Requirement already satisfied: xgboost==2.1.3 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.1.3)
Collecting huggingface-hub==0.27.0 (from autotrain-advanced)
  Downloading huggingface_hub-0.27.0-py3-none-any.whl.metadata (13 kB)
Requirement already satisfied: requests==2.32.3 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.32.3)
Requirement already satisfied: einops==0.8.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.8.0)
Requirement already satisfied: packaging==24.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (24.2)
Requirement already satisfied: cryptography==44.0.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (44.0.0)
Requirement already satisfied: nvitop==1.3.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.3.2)
Requirement already satisfied: tensorboard==2.18.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.18.0)
Requirement already satisfied: peft==0.14.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.14.0)
Requirement already satisfied: trl==0.13.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.13.0)
Requirement already satisfied: tiktoken==0.8.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.8.0)
Collecting transformers==4.48.0 (from autotrain-advanced)
  Downloading transformers-4.48.0-py3-none-any.whl.metadata (44 kB)
Requirement already satisfied: accelerate==1.2.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.2.1)
Requirement already satisfied: rouge-score==0.1.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.1.2)
Requirement already satisfied: py7zr==0.22.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.22.0)
Requirement already satisfied: fastapi==0.115.6 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.115.6)
Requirement already satisfied: uvicorn==0.34.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.34.0)
Requirement already satisfied: python-multipart==0.0.20 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.0.20)
Requirement already satisfied: pydantic==2.10.4 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.10.4)
Requirement already satisfied: hf-transfer in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.1.9)
Requirement already satisfied: pyngrok==7.2.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (7.2.1)
Requirement already satisfied: authlib==1.4.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.4.0)
Requirement already satisfied: itsdangerous==2.2.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.2.0)
Requirement already satisfied: seqeval==1.2.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.2.2)
Requirement already satisfied: httpx==0.28.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.28.1)
Requirement already satisfied: pyyaml==6.0.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (6.0.2)
Requirement already satisfied: timm==1.0.12 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.0.12)
Requirement already satisfied: torchmetrics==1.6.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.6.0)
Requirement already satisfied: pycocotools==2.0.8 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.0.8)
Requirement already satisfied: sentence-transformers==3.3.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (3.3.1)
Requirement already satisfied: bitsandbytes==0.45.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.45.0)
Requirement already satisfied: numpy<3.0.0,>=1.17 in ./env/lib/python3.10/site-packages (from accelerate==1.2.1->autotrain-advanced) (2.0.1)
Requirement already satisfied: psutil in ./env/lib/python3.10/site-packages (from accelerate==1.2.1->autotrain-advanced) (6.1.1)
Requirement already satisfied: torch>=1.10.0 in ./env/lib/python3.10/site-packages (from accelerate==1.2.1->autotrain-advanced) (2.4.0)
Requirement already satisfied: safetensors>=0.4.3 in ./env/lib/python3.10/site-packages (from accelerate==1.2.1->autotrain-advanced) (0.5.2)
Requirement already satisfied: scipy>=1.10.0 in ./env/lib/python3.10/site-packages (from albumentations==1.4.23->autotrain-advanced) (1.15.1)
Requirement already satisfied: albucore==0.0.21 in ./env/lib/python3.10/site-packages (from albumentations==1.4.23->autotrain-advanced) (0.0.21)
Requirement already satisfied: eval-type-backport in ./env/lib/python3.10/site-packages (from albumentations==1.4.23->autotrain-advanced) (0.2.2)
Requirement already satisfied: opencv-python-headless>=4.9.0.80 in ./env/lib/python3.10/site-packages (from albumentations==1.4.23->autotrain-advanced) (4.11.0.86)
Requirement already satisfied: stringzilla>=3.10.4 in ./env/lib/python3.10/site-packages (from albucore==0.0.21->albumentations==1.4.23->autotrain-advanced) (3.11.3)
Requirement already satisfied: simsimd>=5.9.2 in ./env/lib/python3.10/site-packages (from albucore==0.0.21->albumentations==1.4.23->autotrain-advanced) (6.2.1)
Requirement already satisfied: typing_extensions>=4.8.0 in ./env/lib/python3.10/site-packages (from bitsandbytes==0.45.0->autotrain-advanced) (4.12.2)
Requirement already satisfied: cffi>=1.12 in ./env/lib/python3.10/site-packages (from cryptography==44.0.0->autotrain-advanced) (1.17.1)
Requirement already satisfied: dill in ./env/lib/python3.10/site-packages (from evaluate==0.4.3->autotrain-advanced) (0.3.8)
Requirement already satisfied: xxhash in ./env/lib/python3.10/site-packages (from evaluate==0.4.3->autotrain-advanced) (3.5.0)
Requirement already satisfied: multiprocess in ./env/lib/python3.10/site-packages (from evaluate==0.4.3->autotrain-advanced) (0.70.16)
Requirement already satisfied: fsspec>=2021.05.0 in ./env/lib/python3.10/site-packages (from fsspec[http]>=2021.05.0->evaluate==0.4.3->autotrain-advanced) (2024.9.0)
Requirement already satisfied: starlette<0.42.0,>=0.40.0 in ./env/lib/python3.10/site-packages (from fastapi==0.115.6->autotrain-advanced) (0.41.3)
Requirement already satisfied: annotated-types>=0.6.0 in ./env/lib/python3.10/site-packages (from pydantic==2.10.4->autotrain-advanced) (0.7.0)
Requirement already satisfied: pydantic-core==2.27.2 in ./env/lib/python3.10/site-packages (from pydantic==2.10.4->autotrain-advanced) (2.27.2)
Requirement already satisfied: anyio in ./env/lib/python3.10/site-packages (from httpx==0.28.1->autotrain-advanced) (4.8.0)
Requirement already satisfied: certifi in ./env/lib/python3.10/site-packages (from httpx==0.28.1->autotrain-advanced) (2024.12.14)
Requirement already satisfied: httpcore==1.* in ./env/lib/python3.10/site-packages (from httpx==0.28.1->autotrain-advanced) (1.0.7)
Requirement already satisfied: idna in ./env/lib/python3.10/site-packages (from httpx==0.28.1->autotrain-advanced) (3.7)
Requirement already satisfied: filelock in ./env/lib/python3.10/site-packages (from huggingface-hub==0.27.0->autotrain-advanced) (3.13.1)
Requirement already satisfied: click<9.0.0,>=8.1.3 in ./env/lib/python3.10/site-packages (from jiwer==3.0.5->autotrain-advanced) (8.1.8)
Requirement already satisfied: rapidfuzz<4,>=3 in ./env/lib/python3.10/site-packages (from jiwer==3.0.5->autotrain-advanced) (3.11.0)
Requirement already satisfied: regex>=2021.8.3 in ./env/lib/python3.10/site-packages (from nltk==3.9.1->autotrain-advanced) (2024.11.6)
Requirement already satisfied: nvidia-ml-py<12.536.0a0,>=11.450.51 in ./env/lib/python3.10/site-packages (from nvitop==1.3.2->autotrain-advanced) (12.535.161)
Requirement already satisfied: cachetools>=1.0.1 in ./env/lib/python3.10/site-packages (from nvitop==1.3.2->autotrain-advanced) (5.5.0)
Requirement already satisfied: termcolor>=1.0.0 in ./env/lib/python3.10/site-packages (from nvitop==1.3.2->autotrain-advanced) (2.5.0)
Requirement already satisfied: alembic>=1.5.0 in ./env/lib/python3.10/site-packages (from optuna==4.1.0->autotrain-advanced) (1.14.1)
Requirement already satisfied: colorlog in ./env/lib/python3.10/site-packages (from optuna==4.1.0->autotrain-advanced) (6.9.0)
Requirement already satisfied: sqlalchemy>=1.4.2 in ./env/lib/python3.10/site-packages (from optuna==4.1.0->autotrain-advanced) (2.0.37)
Requirement already satisfied: python-dateutil>=2.8.2 in ./env/lib/python3.10/site-packages (from pandas==2.2.3->autotrain-advanced) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in ./env/lib/python3.10/site-packages (from pandas==2.2.3->autotrain-advanced) (2024.2)
Requirement already satisfied: tzdata>=2022.7 in ./env/lib/python3.10/site-packages (from pandas==2.2.3->autotrain-advanced) (2024.2)
Requirement already satisfied: texttable in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (1.7.0)
Requirement already satisfied: pycryptodomex>=3.16.0 in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (3.21.0)
Requirement already satisfied: pyzstd>=0.15.9 in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (0.16.2)
Requirement already satisfied: pyppmd<1.2.0,>=1.1.0 in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (1.1.1)
Requirement already satisfied: pybcj<1.1.0,>=1.0.0 in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (1.0.3)
Requirement already satisfied: multivolumefile>=0.2.3 in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (0.2.3)
Requirement already satisfied: inflate64<1.1.0,>=1.0.0 in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (1.0.1)
Requirement already satisfied: brotli>=1.1.0 in ./env/lib/python3.10/site-packages (from py7zr==0.22.0->autotrain-advanced) (1.1.0)
Requirement already satisfied: matplotlib>=2.1.0 in ./env/lib/python3.10/site-packages (from pycocotools==2.0.8->autotrain-advanced) (3.10.0)
Requirement already satisfied: charset-normalizer<4,>=2 in ./env/lib/python3.10/site-packages (from requests==2.32.3->autotrain-advanced) (3.3.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./env/lib/python3.10/site-packages (from requests==2.32.3->autotrain-advanced) (2.3.0)
Requirement already satisfied: absl-py in ./env/lib/python3.10/site-packages (from rouge-score==0.1.2->autotrain-advanced) (2.1.0)
Requirement already satisfied: six>=1.14.0 in ./env/lib/python3.10/site-packages (from rouge-score==0.1.2->autotrain-advanced) (1.17.0)
Requirement already satisfied: threadpoolctl>=3.1.0 in ./env/lib/python3.10/site-packages (from scikit-learn==1.6.0->autotrain-advanced) (3.5.0)
Requirement already satisfied: tokenizers<0.22,>=0.21 in ./env/lib/python3.10/site-packages (from transformers==4.48.0->autotrain-advanced) (0.21.0)
Requirement already satisfied: grpcio>=1.48.2 in ./env/lib/python3.10/site-packages (from tensorboard==2.18.0->autotrain-advanced) (1.69.0)
Requirement already satisfied: markdown>=2.6.8 in ./env/lib/python3.10/site-packages (from tensorboard==2.18.0->autotrain-advanced) (3.7)
Requirement already satisfied: protobuf!=4.24.0,>=3.19.6 in ./env/lib/python3.10/site-packages (from tensorboard==2.18.0->autotrain-advanced) (5.29.3)
Requirement already satisfied: setuptools>=41.0.0 in ./env/lib/python3.10/site-packages (from tensorboard==2.18.0->autotrain-advanced) (75.1.0)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in ./env/lib/python3.10/site-packages (from tensorboard==2.18.0->autotrain-advanced) (0.7.2)
Requirement already satisfied: torchvision in ./env/lib/python3.10/site-packages (from timm==1.0.12->autotrain-advanced) (0.19.0)
Requirement already satisfied: lightning-utilities>=0.8.0 in ./env/lib/python3.10/site-packages (from torchmetrics==1.6.0->autotrain-advanced) (0.11.9)
Requirement already satisfied: rich in ./env/lib/python3.10/site-packages (from trl==0.13.0->autotrain-advanced) (13.9.4)
Requirement already satisfied: h11>=0.8 in ./env/lib/python3.10/site-packages (from uvicorn==0.34.0->autotrain-advanced) (0.14.0)
Requirement already satisfied: MarkupSafe>=2.1.1 in ./env/lib/python3.10/site-packages (from werkzeug==3.1.3->autotrain-advanced) (2.1.3)
Requirement already satisfied: nvidia-nccl-cu12 in ./env/lib/python3.10/site-packages (from xgboost==2.1.3->autotrain-advanced) (2.24.3)
Requirement already satisfied: pyarrow>=15.0.0 in ./env/lib/python3.10/site-packages (from datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (19.0.0)
Requirement already satisfied: aiohttp in ./env/lib/python3.10/site-packages (from datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (3.11.11)
Requirement already satisfied: exceptiongroup>=1.0.2 in ./env/lib/python3.10/site-packages (from anyio->httpx==0.28.1->autotrain-advanced) (1.2.2)
Requirement already satisfied: sniffio>=1.1 in ./env/lib/python3.10/site-packages (from anyio->httpx==0.28.1->autotrain-advanced) (1.3.1)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (2.4.4)
Requirement already satisfied: aiosignal>=1.1.2 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (1.3.2)
Requirement already satisfied: async-timeout<6.0,>=4.0 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (5.0.1)
Requirement already satisfied: attrs>=17.3.0 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (24.3.0)
Requirement already satisfied: frozenlist>=1.1.1 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (1.5.0)
Requirement already satisfied: multidict<7.0,>=4.5 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (6.1.0)
Requirement already satisfied: propcache>=0.2.0 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (0.2.1)
Requirement already satisfied: yarl<2.0,>=1.17.0 in ./env/lib/python3.10/site-packages (from aiohttp->datasets~=3.2.0->datasets[vision]~=3.2.0->autotrain-advanced) (1.18.3)
Requirement already satisfied: Mako in ./env/lib/python3.10/site-packages (from alembic>=1.5.0->optuna==4.1.0->autotrain-advanced) (1.3.8)
Requirement already satisfied: pycparser in ./env/lib/python3.10/site-packages (from cffi>=1.12->cryptography==44.0.0->autotrain-advanced) (2.22)
Requirement already satisfied: contourpy>=1.0.1 in ./env/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools==2.0.8->autotrain-advanced) (1.3.1)
Requirement already satisfied: cycler>=0.10 in ./env/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools==2.0.8->autotrain-advanced) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in ./env/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools==2.0.8->autotrain-advanced) (4.55.3)
Requirement already satisfied: kiwisolver>=1.3.1 in ./env/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools==2.0.8->autotrain-advanced) (1.4.8)
Requirement already satisfied: pyparsing>=2.3.1 in ./env/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools==2.0.8->autotrain-advanced) (3.2.1)
Requirement already satisfied: greenlet!=0.4.17 in ./env/lib/python3.10/site-packages (from sqlalchemy>=1.4.2->optuna==4.1.0->autotrain-advanced) (3.1.1)
Requirement already satisfied: sympy in ./env/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==1.2.1->autotrain-advanced) (1.13.3)
Requirement already satisfied: networkx in ./env/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==1.2.1->autotrain-advanced) (3.4.2)
Requirement already satisfied: jinja2 in ./env/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==1.2.1->autotrain-advanced) (3.1.4)
Requirement already satisfied: markdown-it-py>=2.2.0 in ./env/lib/python3.10/site-packages (from rich->trl==0.13.0->autotrain-advanced) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in ./env/lib/python3.10/site-packages (from rich->trl==0.13.0->autotrain-advanced) (2.19.1)
Requirement already satisfied: mdurl~=0.1 in ./env/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich->trl==0.13.0->autotrain-advanced) (0.1.2)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in ./env/lib/python3.10/site-packages (from sympy->torch>=1.10.0->accelerate==1.2.1->autotrain-advanced) (1.3.0)
Downloading autotrain_advanced-0.8.36-py3-none-any.whl (341 kB)
Downloading huggingface_hub-0.27.0-py3-none-any.whl (450 kB)
Downloading transformers-4.48.0-py3-none-any.whl (9.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.7/9.7 MB 154.1 MB/s eta 0:00:00
Installing collected packages: huggingface-hub, transformers, autotrain-advanced
  Attempting uninstall: huggingface-hub
    Found existing installation: huggingface-hub 0.32.4
    Uninstalling huggingface-hub-0.32.4:
      Successfully uninstalled huggingface-hub-0.32.4
  Attempting uninstall: transformers
    Found existing installation: transformers 4.53.0.dev0
    Uninstalling transformers-4.53.0.dev0:
      Successfully uninstalled transformers-4.53.0.dev0

Successfully installed autotrain-advanced-0.8.36 huggingface-hub-0.27.0 transformers-4.48.0
INFO     | 2025-06-07 05:43:51 | autotrain.app.ui_routes:<module>:31 - Starting AutoTrain...
INFO     | 2025-06-07 05:43:52 | autotrain.app.ui_routes:<module>:315 - AutoTrain started successfully
INFO     | 2025-06-07 05:43:52 | autotrain.app.app:<module>:13 - Starting AutoTrain...
INFO     | 2025-06-07 05:43:53 | autotrain.app.app:<module>:23 - AutoTrain version: 0.8.36
INFO     | 2025-06-07 05:43:53 | autotrain.app.app:<module>:24 - AutoTrain started successfully
INFO:     Started server process [95]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:7860 (Press CTRL+C to quit)
INFO:     10.16.13.110:31149 - "GET /?logs=container&__sign=eyJhbGciOiJFZERTQSJ9.eyJyZWFkIjp0cnVlLCJwZXJtaXNzaW9ucyI6eyJyZXBvLmNvbnRlbnQucmVhZCI6dHJ1ZX0sIm9uQmVoYWxmT2YiOnsia2luZCI6InVzZXIiLCJfaWQiOiI2N2RjZWRlNTQwMDRiMmYwZGVmYWRiN2EiLCJ1c2VyIjoia2FpZG9tb3RvIiwic2Vzc2lvbklkIjoiNjgzYzQ3NzIxNzEzODZhMGY5MDljZmE4In0sImlhdCI6MTc0OTI3NTAzNSwic3ViIjoiL3NwYWNlcy9rYWlkb21vdG8vcXdlbjNfOGJfbG9yYV9kZW1vIiwiZXhwIjoxNzQ5MzYxNDM1LCJpc3MiOiJodHRwczovL2h1Z2dpbmdmYWNlLmNvIn0.xtiqcDXOJn6ZirYflcOQPyCi1m8G-opTkTpJU4MTavc9hR47AfW6HFmvMMIkjjmm1Zi0E-bC0jkyRdOYUEUXCQ HTTP/1.1" 307 Temporary Redirect
INFO:     10.16.39.153:59447 - "GET /?logs=container&__sign=eyJhbGciOiJFZERTQSJ9.eyJyZWFkIjp0cnVlLCJwZXJtaXNzaW9ucyI6eyJyZXBvLmNvbnRlbnQucmVhZCI6dHJ1ZX0sIm9uQmVoYWxmT2YiOnsia2luZCI6InVzZXIiLCJfaWQiOiI2N2RjZWRlNTQwMDRiMmYwZGVmYWRiN2EiLCJ1c2VyIjoia2FpZG9tb3RvIiwic2Vzc2lvbklkIjoiNjgzYzQ3NzIxNzEzODZhMGY5MDljZmE4In0sImlhdCI6MTc0OTI3NTAzNSwic3ViIjoiL3NwYWNlcy9rYWlkb21vdG8vcXdlbjNfOGJfbG9yYV9kZW1vIiwiZXhwIjoxNzQ5MzYxNDM1LCJpc3MiOiJodHRwczovL2h1Z2dpbmdmYWNlLmNvIn0.xtiqcDXOJn6ZirYflcOQPyCi1m8G-opTkTpJU4MTavc9hR47AfW6HFmvMMIkjjmm1Zi0E-bC0jkyRdOYUEUXCQ HTTP/1.1" 307 Temporary Redirect
INFO:     10.16.39.153:59447 - "GET /ui/?logs=container&__sign=eyJhbGciOiJFZERTQSJ9.eyJyZWFkIjp0cnVlLCJwZXJtaXNzaW9ucyI6eyJyZXBvLmNvbnRlbnQucmVhZCI6dHJ1ZX0sIm9uQmVoYWxmT2YiOnsia2luZCI6InVzZXIiLCJfaWQiOiI2N2RjZWRlNTQwMDRiMmYwZGVmYWRiN2EiLCJ1c2VyIjoia2FpZG9tb3RvIiwic2Vzc2lvbklkIjoiNjgzYzQ3NzIxNzEzODZhMGY5MDljZmE4In0sImlhdCI6MTc0OTI3NTAzNSwic3ViIjoiL3NwYWNlcy9rYWlkb21vdG8vcXdlbjNfOGJfbG9yYV9kZW1vIiwiZXhwIjoxNzQ5MzYxNDM1LCJpc3MiOiJodHRwczovL2h1Z2dpbmdmYWNlLmNvIn0.xtiqcDXOJn6ZirYflcOQPyCi1m8G-opTkTpJU4MTavc9hR47AfW6HFmvMMIkjjmm1Zi0E-bC0jkyRdOYUEUXCQ HTTP/1.1" 200 OK
INFO:     10.16.39.72:62050 - "GET /static/scripts/fetch_data_and_update_models.js?cb=2025-06-07%2005:43:55 HTTP/1.1" 200 OK
INFO:     10.16.10.84:42614 - "GET /static/scripts/poll.js?cb=2025-06-07%2005:43:55 HTTP/1.1" 200 OK
INFO:     10.16.39.153:59447 - "GET /static/scripts/listeners.js?cb=2025-06-07%2005:43:55 HTTP/1.1" 200 OK
INFO:     10.16.13.110:31149 - "GET /static/scripts/utils.js?cb=2025-06-07%2005:43:55 HTTP/1.1" 200 OK
INFO:     10.16.39.153:59447 - "GET /static/scripts/logs.js?cb=2025-06-07%2005:43:55 HTTP/1.1" 200 OK
INFO:     10.16.10.84:42614 - "GET /ui/is_model_training HTTP/1.1" 200 OK
INFO:     10.16.13.110:31149 - "GET /ui/model_choices/llm%3Asft HTTP/1.1" 200 OK
INFO     | 2025-06-07 05:43:56 | autotrain.app.ui_routes:fetch_params:415 - Task: llm:sft
INFO:     10.16.39.72:62050 - "GET /ui/params/llm%3Asft/basic HTTP/1.1" 200 OK
INFO:     10.16.39.153:59447 - "GET /ui/accelerators HTTP/1.1" 200 OK
 

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Request for Refund and Support: AutoTrain Space Failed, Unexpected Charges, and Billing Status Issue #895

==========
== CUDA ==

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Request for Refund and Support: AutoTrain Space Failed, Unexpected Charges, and Billing Status Issue #895

Description

========== == CUDA ==

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

==========
== CUDA ==