Exception Access Denied for amazon s3 while Using pre-trained embeddings example.

[Zsclarx]

---

💥 Issue: AccessDenied When Using pretrained() in Spark NLP Standalone Scala App

❓ Problem Description

I’m using Spark NLP version 6.0.3 in a Scala (2.12) Spark application (Spark 3.5.5), and I encounter an S3 Access Denied error when trying to load a pretrained model using the .pretrained() method, specifically with AlbertEmbeddings.

import org.apache.spark.sql.SparkSession
import com.johnsnowlabs.nlp.base.DocumentAssembler
import com.johnsnowlabs.nlp.annotators.Tokenizer
import com.johnsnowlabs.nlp.embeddings.AlbertEmbeddings
import com.johnsnowlabs.nlp.EmbeddingsFinisher
import org.apache.spark.ml.Pipeline

object EmbeddingExample {

  def main(args: Array[String]): Unit = {
    // Create Spark session
    val spark = SparkSession.builder()
//      .appName("Spark NLP Embedding Example")
//      .master("local[*]") // Use all cores locally
//      .config("spark.driver.memory", "4G")
//      .config("spark.kryoserializer.buffer.max", "1000M")
//      .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
//      .getOrCreate()

      .master("local[*]")
      .config("spark.driver.memory", "16G")
      .config("spark.driver.maxResultSize", "0")
      .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .config("spark.kryoserializer.buffer.max", "2000m")
      .config("spark.jsl.settings.pretrained.cache_folder", "./sample_data/pretrained")
      .config("spark.jsl.settings.storage.cluster_tmp_dir", "./sample_data/storage")
      .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:6.0.1")
      .getOrCreate()


    import spark.implicits._

    // NLP pipeline stages
    val documentAssembler = new DocumentAssembler()
      .setInputCol("text")
      .setOutputCol("document")

    val tokenizer = new Tokenizer()
      .setInputCols("document")
      .setOutputCol("token")

    val embeddings = AlbertEmbeddings.pretrained("albert_base_uncased", "en", "tf")
      .setInputCols("token", "document")
      .setOutputCol("embeddings")

    val embeddingsFinisher = new EmbeddingsFinisher()
      .setInputCols("embeddings")
      .setOutputCols("finished_embeddings")
      .setOutputAsVector(true)
      .setCleanAnnotations(false)

    val pipeline = new Pipeline().setStages(Array(
      documentAssembler,
      tokenizer,
      embeddings,
      embeddingsFinisher
    ))

    // Sample data
    val data = Seq("This is a sentence.").toDF("text")

    // Transform
    val result = pipeline.fit(data).transform(data)

    // Show results
    result.selectExpr("explode(finished_embeddings) as result").show(5, 80)

    spark.stop()
  }
}

On execution, I receive the following error:

Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied 
(Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; 
Request ID: 5HBA3PB5G6TXK9M1; 
S3 Extended Request ID: RHTyx7ZM7drntF9nbFASxnO2Gt8YXpNENWG7K8Z1xgQeu180eeDKru9apX/nhB5S0NTiRTfctzU=)

This appears to occur when .pretrained(...) attempts to download model files from a remote S3 bucket maintained by John Snow Labs.

---

🧱 Reproducible Code

val embeddings = AlbertEmbeddings.pretrained("albert_base_uncased", "en", "tf")

Full pipeline setup includes:

• DocumentAssembler
• Tokenizer
• AlbertEmbeddings
• EmbeddingsFinisher

Running on local machine (master("local[*]")).

---

⚙️ Environment Setup

• Spark NLP version: 6.0.3
• Scala version: 2.12.18
• Spark version: 3.5.5
• Build tool: sbt
• Platform: Standalone (non-Notebook), Local development

Relevant build.sbt configuration:

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-sql_2.12" % "3.5.6",
  "com.johnsnowlabs.nlp" %% "spark-nlp" % "6.0.3"
)

Resolvers:

resolvers ++= Seq(
  "Maven Central" at "https://repo1.maven.org/maven2",
  "Spark Packages" at "https://repos.spark-packages.org"
)

---

🔍 Additional Notes

• When I omit the engine = "tf" parameter in .pretrained(), it defaults to OpenVINO engine and attempts to download incompatible binaries.
• Ideally, I want to use ONNX weights with this embedding if possible.
• I have not configured any AWS credentials, nor do I want to provide them for public models.

---

✅ Request for Help

• Is there a way to resolve this S3 AccessDenied issue when using .pretrained() in a local Spark app?
• What is the correct way to use ONNX-based embeddings in Spark NLP with Scala?
• Is manual model download and loading via load() the only option here?
• Are there any offline/airgapped deployment instructions or pre-downloaded model bundles available?

---

Thanks in advance for any insights!

[DevinTDHa]

Hi @Zsclarx,

It seems like you did not use the right syntax to download our pretrained models. The third argument of the pretrained function specifies the folder to download the model from, and we don't use a "tf" folder. Our tf/onnx/openvino models share the same folder (which is "public/models"). You don't need to specify this usually, except for custom setups. Also, the default language is already "en" so you don't need to specify this either.

So I suggest trying this:

val embeddings = AlbertEmbeddings
  .pretrained("albert_base_uncased") // Note the removed arguments
  .setInputCols("sentence", "token")
  .setOutputCol("embeddings")

Let me know if that helps.