Not a Member? Got you here

Step 1: Install Homebrew (if you haven’t already)

Homebrew is a package manager for macOS that simplifies the installation of software. If you haven’t installed it yet, open your terminal and run:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Step 2: Install Python (if you don’t have it)

It’s recommended that you use the latest version of Python. You can install it via Homebrew:

brew install python

Step 3: Set up a Virtual Environment (optional but recommended)

Creating a virtual environment helps manage dependencies for your projects. You can create one using venv:

python3 -m venv transformers-env
source transformers-env/bin/activate

Step 4: Install Rust Using curl

Open your terminal and run the following command:

1. curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

2. source $HOME/.cargo/env

The installation can be verified by:

rustc --version

Step 5: Install PyTorch

The transformers library often relies on PyTorch. You can install the version optimized for Apple Silicon (ARM architecture) using the following command:

pip install torch --extra-index-url https://download.pytorch.org/whl/metal.html

Step 6: Upgrade pip and setuptools

Make sure you have the latest version of pip and setuptools:

pip install --upgrade pip setuptools

Step 7: Install Transformers and Tokenizers

Now you can install the transformers or tokenizers libraries:

pip install transformers 
pip install tokenizers
Installing transformers/tokenizers on Apple Silicon Mac (M1/M2): A Step-by-Step Guide that works
Successful Installation

Conclusion/Additional Steps for Debugging:

Sometimes there is a problem with the compatibility of Python’s older versions and `tokenizers`. The fact that after a while, the Rust compiler became more strict is the reason behind this.

You might see an error along the lines of:

error: cargo rustc — lib — message-format=json-render-diagnostics — manifest-path Cargo.toml — release -v — features pyo3/extension-module — crate-type cdylib — -C ‘link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/tokenizers.cpython-37m-darwin.so’ failed with code 101

which can be solved by:

export RUSTFLAGS="-A invalid_reference_casting"