What to Study to Learn How to Build Integrations Between Systems Using Apache Camel
Apache Camel is an integration framework, which basically means that it is a ready of tools that will help yous connect i system to another. A basic example could be listening to an AWS SQS queue and saving a file in Google Cloud Storage Saucepan for each message that has been received.
It comes already with an implementation of the most used patterns divers in the book "Enterprise Integration Patterns", which we could brand a similarity with the book of Pattern Patterns by "the Gang of Four" but at the level of systems integration. A volume that defines a series of patterns and blueprints to help united states of america to amend design big component-based applications.
Why is an integration framework necessary?
Clearly, Apache Camel (every bit any other integration framework) is a solution to a problem – either an essential or an accidental ane. In microservices oriented applications, the communication between them can be of many types, as discussed in DDD's "strategic pattern", at that place are many ways to communicate between divisional contexts (microservices), for example, the mapping of "Conformist" type:
It tells us that the downstream service (D) conforms to the data that comes to information technology and as it comes to it, it has no choice since the ability lies with the upstream service (U).
Take information technology or leave it
This type of communication is an example that we will have to adjust to any communication (integration) that nosotros have with this service, sometimes we will have some power to define this communication channel and sometimes non, in others there will be no room for definition considering there already was in its day, legacy projects where there are systems that accept been running for many years and are "untouchable".
Running Camel
Camel can be executed in multiple ways, from standalone, every bit office of springboot or quarkus, or fifty-fifty equally role of kubernetes (with Camel Yard), although it is non express to these, it tin e'er exist extended more than hands to more systems.
Enterprise Integration Patterns (EIP)
It is rare that when you wait for information about Apache Camel you practise not observe the acronym EIP, and that is because Apache Camel was adult as an implementation of the patterns described in the book Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf. Comparable to the Gang of Four's Design Patterns book but at the systems integration level.
To give some examples and to know what we are talking near, here is a sample of some EIPs (you will find all of them in EIP and in the Apache Camel documentation):
Content Based Router
It will assist us to route our incoming message to different destinations according to its content.
from("directly:in") .choice() .when(header("blazon").isEqualTo("widget")) .to("directly:widget") .when(header("blazon").isEqualTo("gadget")) .to("direct:gadget") .otherwise() .to("straight:other"); Filter
With it we volition filter the letters nosotros are really interested in and discard the residue.
from("direct:a") .filter(simple("${header.foo} == 'bar'")) .to("direct:b"); Enricher
When the message we receive does not take all the data we need, with this pattern we tin can look for the missing information in another data source and add it to the original bulletin.
from("straight:kickoff") .enrich("direct:resource", aggregationStrategy) .to("direct:result"); There are two Enrich's strategies in Camel, we will become deeper into them in futurity manufactures.
What Camel offers?
We cannot reduce Apache Camel to an implementation of the EIPs (although a good part of its ability comes from information technology), merely information technology besides provides a broad list of Components (more than 300) to connect with, either equally source or destination of data, since at the end, in Camel we will be developing Routes where we volition connect a Component A with Component B, for instance:
from("aws-sqs://queue-proper noun") to("aws-s3://bucket-name") It is possible that ii components that must communicate with each other practice not speak the same "language" (format), it is possible that the output of one is XML while the input of the other is a JSON, for that camel allows yous to configure a series of Data Format inside its runtime to exist able to convert from one type to another easily.
Not only the format is relevant in the communication, in that location are also different types of data, you can receive a message in JSON format from a queue and want to persist that message in a file in a s3 bucket, Apache Camel volition attempt to catechumen that same body that it receives as JSON in file automatically (as long as it has the necessary component to do information technology, since Camel is not a single package, it usually has a package for each component).
When to use Camel?
This question has been in my heed since I created my first route in Apache Camel. It is piece of cake to come across that when a project has many integrations, Apache Camel volition assist us, it will manage a lot of complication and will unify the way in which nosotros integrate, since for Camel everything is considered a Bulletin, regardless of the data source.
The fact that Camel standardizes all data in the same way will simplify our integrations with other systems. At the routing level, the blazon of data that travels when you lot read a file in an S3 bucket or when y'all read a bulletin in a queue is the same, an Exchange with the Message in it.
Since the decision is very subjective and it will depend a lot on the project and the squad, I will leave here some points to accept into account that tin aid to make the decision whether to apply Apache Camel or not:
- How many sources/destinations your application has?
- How many protocols it speaks?
- What kind of advice (at Mapping Context level) exists between the different bounded contexts? If there is a fluid communication, where you piece of work as a squad, information technology is possible to unify the protocol, format and so on (although information technology is somewhat utopian).
Infrastructure or business?
This other question arises once you already utilise Apache Camel in your application, and it is where to put the logic, if as role of the Camel road, or in the destination of the route? Should I apply the Splitter pattern plus the Content-Based Router inside the Camel route to then ship it to i endpoint or another in the destination, or should I create a single endpoint in the destination that will take care of splitting it and do what it has to practice? Should I enrich the bulletin earlier it reaches the destination or afterward?
These are difficult questions to answer, only I would try to follow the concept of Smart endpoints and dumb pipes, which means that the logic should be mostly at the destination, avoiding as much as possible to put concern logic (rules and invariants that cover the need of the end user) within the routes.
Endpoints
The concept of Endpoint is not a concept sectional to Apache Camel. When we talk about Endpoint, we are referring to a connection point to a service, equally for example it could be an HTTP endpoint, a subscriber in an SNS topic, etc.
At Apache Camel level, an Endpoint is an interface where we will create the Consumer or Producer (or both) from where to get-go/stop a route.
Continuing with the beginning instance: aws-sqs://queue-name is an Endpoint, and since we are calling it from the from(), it requires the Component aws-sqs to be of type Consumer. The aforementioned happens with the .to() method, we could utilize the aforementioned Endpoint equally long as the component is likewise Producer. (To know if a component is Consumer or Producer is in the get-go line of the component documentation).
Why "Smart Endpoints and dumb pipes"?
Information technology would exist logical to retrieve (and this is defined in some manufactures on the Internet), that pipes should only redirect the output message from one betoken to the input of another. A good example of this type of pipes are those of Unix systems: ls | grep dir, where the stdout of the command ls, is redirected to the stdin of grep, the pipe does not accept more work than that, the residue is done by the endpoints.
If nosotros continue with the Unix instance, information technology is true that the pipe just does i thing, but the protocol that all the endpoints follow is the same, stdout -> stdin. In a large application, it is more than likely that we volition detect several protocols in play (sqs -> http, kafka -> ftp, and a long etc.), but this is something that nosotros solve with the components that Camel brings, it is something that does not affect the concept of dumb pipes.
Yet, the format of the data in Unix while executing our programs can change, it tin can exist that the stdout of the data source is JSON while the input of the destination we are looking for is a elementary string (an id, for example). In Unix we would do something like scroll http://product/list | jq .id | http://product/annals (pseudo-code), although it is truthful that the pipes are express to motion the message from one side to another, in the consummate path nosotros have added a program that transforms the output of our information source and adapts it to the input of our destination.
It is in this type of cases where Camel volition help u.s. without breaking this principle (existence strict nosotros are breaking it… just not everything is black and white).
Another very unlike thing would be, following the previous example, to add in the route a check that the product has stock in the warehouse before registering it, this is an invariant that we could put previously earlier registering the product, but and then we would be separating our business organisation logic to register a product in two different services, with all the bug that this entails.
In my stance, endpoints understand business concepts, they speak the business language (ubiquitous language) – Invariants, Business Rules, business flows, etc. While at the impaired pipes level, if I see well to be able to have some infrastructure logic every bit could be a filtering of some unwanted message past the destination endpoint, transformation of message format (XML to JSON), normalize messages arriving from diverse systems (sqs, kafka, rabbitmq) removing metadata that includes each system and a long etcetera that we will discover, equally you see, closely related to infrastructure bug.
Conclusions
Camel has a medium learning bend, it is not difficult to beginning developing robust routes that can achieve production, merely information technology will be circuitous to chief the whole arrangement that is mounted, at that place are always details that make you not stop learning (and block y'all sometimes in your evolution).
Nor will it prevent you lot from having to read the documentation for any component, if for instance yous are using aws-s3 (either for source or target), if nosotros read the implementation, we volition see that it is using the AWS SDK then zilch will avoid having to know it, all the configuration properties you lot will detect them in the Camel documentation but also in the AWS documentation. Comparable to wanting to use Terraform without knowing AWS, GCP or Azure, or wanting to employ TailwindCSS without knowing CSS.
This commodity is the beginning of a serial where we will go into more detail, until we become to a point of talking most some specific components, with examples of employ and best practices.
Source: https://apiumhub.com/tech-blog-barcelona/apache-camel-101/
0 Response to "What to Study to Learn How to Build Integrations Between Systems Using Apache Camel"
Post a Comment