In a project I am working on we integrate the different products with kafka and use avro. And to facilitate the generation of avro entities in java with their attributes and methods we use the avro-maven-plugin. This means that we have several typed entities that compose a typed message and that are reused in the different final messages. For example, in the case of an invoice reply message, this message consists of 5 message subtypes:
- metadata.avsc
{
"namespace": "es.feitam.typedmessages",
"type" : "record",
"name" : "Metadata",
"fields" : [
{
"name" : "traceId",
"type" : "string"
}, {
"name" : "applicationProducer",
"type" : "string"
}, {
"name" : "instant",
"type" : {
"type": "long",
"logicalType" : "timestamp-millis"
}
}
]
}
- error.avsc
{
"type": "record",
"namespace": "es.feitam.typedmessages",
"name": "Error",
"fields": [
{ "name": "errorCode", "type": "string", "doc" : "Error code" },
{ "name": "errorDescription", "type": "string" , "doc" : "Error description"},
{ "name": "errorLevel", "type": "string" , "doc" : "Error level: E (Error) | W (warning)"}
]
}
- invoiceDetail.avsc
{
"type": "record",
"namespace": "es.feitam.typedmessages",
"name": "InvoiceDetail",
"fields": [
{ "name": "detailId", "type": "int", "doc": "Detail Id" },
{ "name": "detailDescription", "type": "string" , "doc": "Detail description"},
{ "name": "detailAmount", "type": "double" , "doc": "Detail amount"}
]
}
- responseInvoiceContent.avsc
{
"type": "record",
"namespace": "es.feitam.typedmessages",
"name": "ResponseInvoiceContent",
"fields": [
{ "name": "invoiceNumber", "type": "string" , "doc" : "Invoice number"},
{ "name": "invoiceDescription", "type": "string" , "doc" : "Invoice description"},
{ "name": "invoiceDetails", "type": { "type": "array", "items": "InvoiceDetail"}, "doc": "Invoice detail list" },
{ "name": "errors", "type": { "type": "array", "items": "Error"}, "doc": "Error message list" }
]
}
- responseInvoice.avsc
{
"type": "record",
"namespace": "es.feitam.typedmessages",
"name": "ResponseInvoice",
"fields": [
{ "name": "header", "type": "Metadata" , "doc": "Header block" },
{ "name": "body", "type": "ResponseInvoiceContent", "doc": "Body block" }
]
}
In that case the responseInvoice.avsc message is composed of metadata.avsc and responseInvoiceContent.avsc. And responseInvoiceContent.avsc consists of an array of type invoiceDetail.avsc and an array of type error.avsc.
As there are dependencies / relationships between different types if you specify a typical avro-maven-plugin plugin configuration like the following:
...
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>${avro.version}</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>schema</goal>
</goals>
<configuration>
<sourceDirectory>${project.basedir}/src/main/resources/typedmessages/</sourceDirectory>
<includes>
<include>*.avsc</include>
</includes>
<outputDirectory>${project.build.directory}/generated-sources</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
...
This error will be generated:
[ERROR] Failed to execute goal org.apache.avro:avro-maven-plugin:1.9.1:schema (default) on project avromavenplugin-example: Execution default of goal org.apache.avro:avro-maven-plugin:1.9.1:schema failed: "Metadata" is not a defined name. The type of the "header" field must be a defined name or a {"type": ...} expression. -> [Help 1]
This error occurs when trying to generate the class of one of the avro types defined in /src/main/resources/typedmessages/ whose reference Metadata fails because the metadata.avsc class has not yet generated the plugin. To solve it we must indicate to the plugin that avsc files must be processed first, being the important order, with the imports tag as shown in the following example:
...
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>${avro.version}</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>schema</goal>
</goals>
<configuration>
<sourceDirectory>${project.basedir}/src/main/resources/typedmessages/</sourceDirectory>
<imports>
<import>${project.basedir}/src/main/resources/typedmessages/metadata.avsc</import>
<import>${project.basedir}/src/main/resources/typedmessages/error.avsc</import>
<import>${project.basedir}/src/main/resources/typedmessages/invoiceDetail.avsc</import>
<import>${project.basedir}/src/main/resources/typedmessages/responseInvoiceContent.avsc</import>
</imports>
<includes>
<include>*.avsc</include>
</includes>
<outputDirectory>${project.build.directory}/generated-sources</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
...
Con esto la ejecución es correcta:
[operatorfeitam@localhost avromavenplugin-example]$ mvn clean compile package
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building avromavenplugin-example 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ avromavenplugin-example ---
[INFO] Deleting /workspaces/eclipse/feitam/avromavenplugin-example/target
[INFO]
[INFO] --- avro-maven-plugin:1.9.1:schema (default) @ avromavenplugin-example ---
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedentities/metadata.avsc
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedentities/error.avsc
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedentities/invoiceDetail.avsc
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedentities/responseInvoiceContent.avsc
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ avromavenplugin-example ---
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 5 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ avromavenplugin-example ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 6 source files to /workspaces/eclipse/feitam/avromavenplugin-example/target/classes
[INFO]
[INFO] --- avro-maven-plugin:1.9.1:schema (default) @ avromavenplugin-example ---
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedentities/metadata.avsc
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedentities/error.avsc
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedentities/invoiceDetail.avsc
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedentities/responseInvoiceContent.avsc
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ avromavenplugin-example ---
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 5 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ avromavenplugin-example ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ avromavenplugin-example ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /workspaces/eclipse/feitam/avromavenplugin-example/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ avromavenplugin-example ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /workspaces/eclipse/feitam/avromavenplugin-example/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ avromavenplugin-example ---
[INFO] Surefire report directory: /workspaces/eclipse/feitam/avromavenplugin-example/target/surefire-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running es.feitam.lab.AppTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ avromavenplugin-example ---
[INFO] Building jar: /workspaces/eclipse/feitam/avromavenplugin-example/target/avromavenplugin-example-1.0-SNAPSHOT.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.009 s
[INFO] Finished at: 2019-12-19T14:39:48+01:00
[INFO] Final Memory: 21M/289M
[INFO] ------------------------------------------------------------------------
[operatorfeitam@localhost avromavenplugin-example]$ mvn clean compile package
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building avromavenplugin-example 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ avromavenplugin-example ---
[INFO] Deleting /workspaces/eclipse/feitam/avromavenplugin-example/target
[INFO]
[INFO] --- avro-maven-plugin:1.9.1:schema (default) @ avromavenplugin-example ---
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedmessages/metadata.avsc
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedmessages/error.avsc
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedmessages/invoiceDetail.avsc
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedmessages/responseInvoiceContent.avsc
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ avromavenplugin-example ---
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 5 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ avromavenplugin-example ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 6 source files to /workspaces/eclipse/feitam/avromavenplugin-example/target/classes
[INFO]
[INFO] --- avro-maven-plugin:1.9.1:schema (default) @ avromavenplugin-example ---
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedmessages/metadata.avsc
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedmessages/error.avsc
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedmessages/invoiceDetail.avsc
[INFO] Importing File: /workspaces/eclipse/feitam/avromavenplugin-example/src/main/resources/typedmessages/responseInvoiceContent.avsc
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ avromavenplugin-example ---
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 5 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ avromavenplugin-example ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ avromavenplugin-example ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /workspaces/eclipse/feitam/avromavenplugin-example/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ avromavenplugin-example ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /workspaces/eclipse/feitam/avromavenplugin-example/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ avromavenplugin-example ---
[INFO] Surefire report directory: /workspaces/eclipse/feitam/avromavenplugin-example/target/surefire-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running es.feitam.lab.AppTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ avromavenplugin-example ---
[INFO] Building jar: /workspaces/eclipse/feitam/avromavenplugin-example/target/avromavenplugin-example-1.0-SNAPSHOT.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.585 s
[INFO] Finished at: 2019-12-19T15:29:31+01:00
[INFO] Final Memory: 21M/278M
[INFO] ------------------------------------------------------------------------
[operatorfeitam@localhost avromavenplugin-example]$
And the java de avro classes are generated in generated-sources/es/feitam/typedmessages:
With this avro-maven-plugin we can generate the java classes that allow us to perform all the required management with avro, in addition to serving us to certify the correct implementation of the definition files of these avro schemes, since if they are wrong the plugin will generate error .
Very important, because it is a mistake that I commonly see: the names of the records must always begin with capital letters because it is the name used for each class (if we put them in lower case the first letter the java class is generated with the name starting in lowercase, and therefore does not meet java specifications). Field names must begin in lowercase because they are the attributes of the java class that make up the entities of these messages plus their management methods. I give you a screenshot of the attributes and methods (whitout non public members) of one of these generated classes:
