Understanding Integration Cloud and how to get the most out of your implementation

Category: Definitive Tips Page 1 of 2

Short tips on ICS. Covering things that have been added to ICS since the book was publisahed or couldn’t be fitted in.

Quarterly Updates

The update regime for Integration Cloud is well established in its quarterly pattern, but within that pattern are two update cycles, separated by two weeks. It is possible to choose which cycle your OIC instance update will be executed in. If you don’t specify which cycle then by default you will be put into the second cycle.

For production deployments of OIC that makes a lot of sense. But we would recommend that your non-production instance be part of the 1st update cycle. This allows you two weeks to validate and fix any issues in the event that the upgrade breaks any of your integrations. While that shouldn’t happen if you are exploiting an undocumented behaviour or something reported as a bug there is always a risk.

So the obvious question is how to define which update cycle should be used. For OCI Gen 2 (the majority of users should have migrated to now), the control is achieved by setting a freeform tag on the OIC instance. The tag needs to be called OIC_UPDATE_WINDOW1 (note – if you don’t read the Oracle documentation carefully you could end up omitting the numeral) and the value can be left blank. The tags are set on the OCI view for your OIC instance, which has a tabbed view as you can see below. Once the value is set then the OCI view will show an Updating status – this is not to be confused with the OIC instance being updated with the latest quarterly changes.

All of this shows up in blog (here and script fragment here) and a documentation (here). What is less apparent is the lead time needed for the tag to be in place. This is in the order of 7 or more weeks. This means you need to have your OIC dev instance in place almost a full quarter before the opportunity is available, and spinning up a new OIC instance and expecting it to immediately adopt the latest version during the maintenance window isn’t going to solve any problems.

How to confirm the instance version

The related question is where to look for the version of OIC is running. The information is only provided in the instance console rather then the OCI View of OIC. The version information is available as part of the drop down visible on the question mark icon at the top right of the UI, as the following screenshots show:

We do hope that Oracle will shorten this in the months to come.

Terraforming …

If you’re building your OIC deployment(s) using Terraform, then you could pass a variable into your Terraform module (hence the reference to var. or read from a configuration file in which case you will want a data block and the value becomes data.)

locals {
 updateWindow = (var.use_window1) ? "OIC_UPDATE_WINDOW1" : "--NOT-WINDOW1--"
}

resource "oci_integration_integration_instance" "test_integration_instance" {
    #Required
    compartment_id = var.compartment_id
    display_name = "ExampleOIC"
    integration_instance_type = var.integration_instance_integration_instance_type
    is_byol = false
    message_packs = 1

    consumption_model = var.integration_instance_consumption_model
    custom_endpoint {
        #Required
        hostname = var.integration_instance_custom_endpoint_hostname

        #Optional
        certificate_secret_id = oci_vault_secret.test_secret.id
    }

    freeform_tags = {"${local.updateWindow}"= ""}
    idcs_at = var.integration_instance_idcs_at
    is_file_server_enabled = var.integration_instance_is_file_server_enabled
    is_visual_builder_enabled = var.integration_instance_is_visual_builder_enabled
    network_endpoint_details {
        #Required
        network_endpoint_type = var.integration_instance_network_endpoint_details_network_endpoint_type


        }
        is_integration_vcn_allowlisted = var.integration_instance_network_endpoint_details_is_integration_vcn_allowlisted
    }
    state = var.integration_instance_target_state
}

As you can see in my example I have hardwired more values than the example provided by the Oracle Terraform documentation (here) as it helps show the legal values. Here to keep the declarations simple – I have set a freeform tag regardless, but changed in the local variable value to be used by the freeform tag depending on if a variable ( use_window1) is set.

OIC Scheduling for End of Month style processes – Tip #11

Setting up calendar schedules that aren’t simply reoccurring, such as a specific date in a month like 28th or third Friday, become more of a challenge with the OIC Scheduler. The calendar works with the ical standard set by the IETF standard 2445. If you review the standard’s section 4.3.0, this describes the reoccurrence options, whilst able, not able to address these needs.

Reoccurring on specific dates can, in theory, be set. If you review section 4.3.4, you will see a date has the potential to be defined as a comma-separated list. So multiple dates for month-end can be provided as a list. Note, this is predicated on the calendar attribute supporting date-mday (DATE). The DTSTART and DTEND attributes accept DATE with no indication that the date-mday is not supported.

Specifying the dates explicitly overcomes the problems of national holidays. The downside is that you need to know when the list needs to be extended and, in the case of OIC, the means to edit the scheduler format safely.

NOTE: we have not tested whether the OIC scheduler is compliant with the use of date-mday at the time of writing.

Versions 1 and 2 running concurrently

Complex formulas for scheduling and more manageable scheduling..

There are several options for managing reoccurring events, such as month-end, year-end accounting, payroll and expenses payments etc.

In most options, we assume that we have a simple reoccurring schedule that triggers an integration that determines whether the trigger is legitimate. This could be as simple as daily. If valid, then invoking the main integration with the business process.

Defining Dates in the database, we can wrap it with JET, Visual Builder or APEX to create a presentation layer. The task of managing the schedule to be taken on by the business.

This approach makes it very easy to adjust and test. Whether there are any more future schedule dates can also be overcome by having an integration that queries the table for the next date being set. If not date set, then the generate relevant alerts using OCI native tools.

The schedule is triggered regularly, e.g. daily, and the scheduler trigger invokes an integration then looks up the DB to see if the date appears in the list. When the date appears, then the integration is executed. It would be straightforward to extend this to trigger different schedules.

Automated option

If an algorithm can be described for when the month-end processes should be performed, we eliminate the need to manage a list of dates. There are some ways to implement a formula, and you have the option of trying to do it OIC, but using OCI Functions – the formula is coded in the language of your choice and then packaged and deployed. The Function is then invoked using a REST web service to confirm whether the process should run (more on how here).

Handling public and national holidays. Not all national holidays are locked into an algorithm, or the holiday is fairly complex. That said, there are several API sources available that can tell you these dates, for example, https://holidayapi.com; this makes things easier.

The beauty of this process is that the same Function could keep a company website up to date with the next run date. Be integrated into corporate notifications and so on. As a result, one point of truth.

Enhanced Automation (feature wish …)

Unfortunately, OIC doesn’t provide an API to add new schedules. We can’t simplify the integration logic until it does, which adds the next calculated date for the process to run as a one-off schedule.

iCal Resources (inc Tools)

Beating OIC’s FTP File Limits

OIC has for some time now provided an FTP adaptor and more recently included a full FTP server capability. But both have limits on the file size and capacity. The file constraints (1GB for a file and 500GB for the FTP server) shouldn’t be an issue for day to day activities. But OIC is often used to support SaaS Financials and other cloud solutions which do have monthly process cycles which can generate significant data volumes, for example, payroll data. The question is how to handle such data with such constraints?

Data Integration?

There is a school of thought that points to the possibility when handling such large data volumes we should consider using Data Integration rather than a more event-centric integration tool. Personally, I think there is a lot of validity in the argument, and anyone dealing with such bulky data activities should review and question if it is a better answer.

That said, there are cases where it does stand-up. For example:

  • If an organization is transitioning to a more event-driven or at least micro-batch model, you have to start the transition somewhere, but trying to line up changes everywhere can be problematic, so we have to start somewhere. Building an integration process so you have an event model developed, but in the interim, you need to take that bulk mechanism and convert it to a small stream of events.
  • You may be working with a bulk data extract and only need a small subset of the data provided, it won’t help if the data is also represented using a verbose notation such as XML.

Other Approaches

How to overcome the constraint? Oracle databases aren’t so constrained, and SQLLoader can provide an easy means to ingest the data into a staging table. The benefit of this is:

  • if you’re only needing a subset of the data you can pull just those columns from the table.
  • the bulk of the use of XML to be self-describing can be shed through using the DB schema as being more prescriptive.
  • SQL scripts can handle the checksum records removing that data and overhead from the integration process, leaving you to concentrate more on the business process.

If the data is still substantial once in the database there are a number of strategies to consume the data in more manageable chunks, such as

  • Running SQL script on the database that takes each row and calls the OIC as a restful API point. This approach is potentially very interesting as it may then mean if you’re moving towards an event process in the future the API endpoint represents the future state and the database stored procedure is mimicking the future client behaviour.
  • Use polling strategies and result set limits to control how much data is processed in a single execution of the integration. This approach does mean the integration needs to tag which records have been processed to avoid re-reading them.

Useful Links

Integration with APIless Systems – Leveraging RPA – Definitive Tip #10

Traditionally integrating with systems that don’t offer APIs or a shared storage mechanism (such as open tables) has been something of a headache often resulting in the ‘last mile’ of the integration process being manual. The manual steps often come because the cost of building and maintaining the means to integrate has not been cost effective or even an option (vendor has end of lifted a product, and not willing to add an integration mechanism).

The idea of ‘screen scraping’ isn’t new, but the cost of implementing such mechanisms has dropped and the new generation of Robotic Process Automation (RPA) tools such those provided by UiPath have made it significantly easier to integrate and automate UI driven processes. Whilst UI based integration isn’t recommended as a first option for integration, it shouldn’t be ruled out particularly as it gets easier and easier to generate and maintain the UI automation. There are several factors that need to be considered as to whether such an approach is appropriate, for example:

  • the ability to run a robot to execute the UI interaction,
  • the volume of data needing to be moved through the UI – you wont escape the latency issues that may exist with UI steps,
  • is the UI being automated changing rapidly (is there enough cost benefit for automating)

1Oracle have been working in partnership with one of the leading RPA product vendors – UiPath, which has resulted in an adaptor for Oracle Integration Cloud.  The adaptor allows you to pass data to the UiPath Orchestrator component which will run the processes in an unattended mode.  In the adaptor configuration you provide information about how many resources you want the Orchestrator to apply to the task, the queuing of the job and so on.

UIP_4698_WEBSITE-GRAPHICS_WEB_100518

UiPath’s Architecture

for more information on RPA and the adaptor the following links maybe of help:

Whilst Oracle’s roadmap in the RPA space is not entirely clear we have heard indications that Oracle are limiting themselves to just UiPath (this is what UiPath say about the partnership).

Regardless of the approach you’ll see that the adoption of RPA is important in Oracle’s vision, with their Agile Finance making a clear indication of its view (see the paper here).

Deploying OIC – Definitive Tip #9

cloud-stack_CloudStack-cloud-270When ICS was launched it delivered on of the values of PaaS namely you didn’t need to worry about setting up storage, database and compute, it was all sorted out for you. Admittedly it come with a lack of elasticity when it came to resilience and scaling. In contrast Services such as SOA CS which required you to go through each of the layers, but gave you a degree of flexibility. The whilst simpler than building SOA on-premises it still represents a laborious and fiddly process that took time.

When OIC (Oracle Integration Cloud) arrived and the introduction Universal Credits we had a pricing model that made it a lot easier to be elastic in terms of approach to resourcing, but a deployment model that following SOA CS rather than ICS. To an extend, one step forward, and another back.

Fortunately we are seeing head way that means we have recovered that backward step.

Using REPL to Simplify ICS Mgmt Tasks – Definitive Tip 10

The REPL tool was built against ICS running on OCI Classic. As this is no longer available the tool has been withdrawn

Oracle A Team

Earlier in 2017, the Oracle A-Team released a Python library that abstracts the ICS REST API to make some management tasks that you may wish to perform easier. For example, identifying a set of integrations and activating or deactivating them, or simply importing and exporting them. In this Definitive Tip, we will look at the REPL tool and its benefits. The REPL tool can be downloaded here.

Getting setup on Windows at least is a little fiddly as you need to install Python and then depending upon your version of Python update the installer and retrieve a number of additional libraries. REPL itself is available here.

ICS Pricing – ICS Definitive Tip #9

Cloud Costs

ICS pricing is based on two aspects – the number of connections and the number of messages processed.  But what constitutes a connection?  What happens if I exceed the number of messages or connections?

Connecting directly to DBaaS – ICS Definitive Tip #7

DBaaSEnriching an integration from data in a database or DBaaS (Database as a Service) is not an unusual requirement. Many integration use cases today need to access a database that is on-premises. The means to connect to the database is fairly obvious – the connection agent. Our book goes into a lot more detail as to why that is, and the implications of using database connections.

However when it comes to Oracle’s DBaaS a service it would be very easy to assume that given that you’re using two different parts of Oracle’s PaaS that it would be straight forward to connect the two together without an agent. However, at least today whether its on-premises or DBaaS you need to use a connection agent. This does mean that you’ll need an IaaS node to host the connection agent.

This quirk is driven by the fact that there are some scenarios that this does actually make sense. For example – the Oracle domains need to have a high level of isolation, so when the DBaaS is in another domain then the decoupling via the agent makes sense. When your database is in a different zone of the cloud – then you’re running DB calls across what is effectively a wide are network – not good.

Conditional Mapping – ICS Definitive Tip #6

The question of conditional mapping comes up regularly on the ICS part of community.oracle.com which prompted us to write this blog. So if you want the output of an integration in element X to contain value from element A or B depending upon element C there are a number of answers, each having pros and cons. So let’s look at them before we offer an example of how we would solve the simpler more common problem.. Your choices are ….

  • Use the power of XSLT within the mapping,
  • Use the conditional option Within a Basic Map Data or an Orchestration pattern,
  • Use a custom piece of JavaScript
XLST Condition in the flow Custom Javascript
Single path through the integration Simpler to read and see visually Can be used either in the condition oer the mapping. Easier to apply than XSLT directly.
ICS doesnt make it the easiest to write this, but reading is fairly easy Easy to visually understand, but means more maintenance effort Easy to incorporate, but makes the development process multistep

We have talked about the condition/filtering in the book, and we will tackle the use of JavaScript in its own Definitive Guide, so let’s look at the XSL mapping option, and probably the route that will cover most cases. In many respects this is nothing to do with ICS, and is all about the use of XSLT so you may find it helpful to have an XSLT reference to hand. To this end the following maybe useful:

  • W3C – perhaps not the easiest to read, but definitely definitive,
  • XMLSchool far more readable but keeps things simple, and focuses on XSLT to create (X)HTML content

Differentiating the ICS Agent Types – Definitive Tip #5

In our book we talked about the difference between the agents offered by ICS, namely the Connection  and Execution agents. Whilst we differentiated the two, we did focus on the connection agent as this is the type we expect to see used in most cases. However the execution agent still suffers from a level of confusion, and it has been helped by being called  ‘ICS on-premises’.

As part of a number of recent conversations the questions and confusion of what the execution agent is and how it works has come up. There is the well known saying ‘a picture is as good as a 1000 words’ which prompted us to develop the diagram below as a power point slide – in its power point form much of the detail is used as an animated build up.

ICS Agent Comparison

Page 1 of 2

Powered by WordPress & Theme by Anders Norén