Spark

JVM since1.3.0 Nativeunsupported

Send RDD or DataFrame jobs to Apache Spark clusters.

What’s inside

Spark component, URI syntax: spark:endpointType

Please refer to the above link for usage and configuration details.

Maven coordinates

<dependency>
    <groupId>org.apache.camel.quarkus</groupId>
    <artifactId>camel-quarkus-spark</artifactId>
</dependency>

Check the User guide for more information about writing Camel Quarkus applications.

Usage

Special BOM

camel-quarkus-spark is special compared to other Camel Quarkus extensions which are managed in camel-quarkus-bom together with their important dependencies. This is because the Camel Spark extension depends on rather old versions of various libraries, that may conflict with their newer versions required by other Quarkus and Camel Quarkus extensions. Therefore, camel-quarkus-spark extension and its dependencies are managed in a special BOM called camel-quarkus-spark-bom.

For camel-quarkus-spark to work properly in end user applications, camel-quarkus-spark-bom need to be imported before any other Quarkus or Camel Quarkus BOMs.

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.apache.camel.quarkus</groupId>
            <artifactId>camel-quarkus-spark-bom</artifactId>
            <version>${camel-quarkus.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>

        <!-- Other BOMs -->
        <dependency>
            <groupId>org.apache.camel.quarkus</groupId>
            <artifactId>camel-quarkus-bom</artifactId>
            <version>${camel-quarkus.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

The above setup will make camel-quarkus-spark work properly but it may break other extensions which are sensitive to downgrades of the transitive dependencies done in camel-quarkus-spark-bom.

Camel Quarkus limitations

spark:dataframe does not work on Java 9+ because Spark 2.x does not support Java 9+. This will be solved when Spark is upgraded to 3.x in Camel, see issue #1955.
spark:hive is does not work, see issue #1956.