A guide to system integration best practices
Following system integration best practices it is possible for organisations to improve efficiency, increase performance and reduce costs. In this article we discuss data integration best practices for enterprise architects including challenges, methods and planning. Download the free digital transformation roadmap as a starting point when planning your integration project.
What is system integration?
System integration is the process of integrating a wide range of business systems, databases and applications together so that data can be pushed and pulled between them.
System integration needs to be able to cater for a wide range of scenarios as business will usually rely on and want to use multiple systems and applications.
These can include large legacy systems, third-party applications, enterprise applications, B2B environments, web services, micro apps and electronic data interchange solutions – whether on-premises or cloud-based.
Once all these are connected, organisations can then automate numerous data exchanges, workflows, notifications and reporting procedures.
When do you need system integration?
Improving business performance is the fundamental reason that many businesses implement system integration.
There are many driving factors for this. It could be that the organisation needs to reduce costs due to the economic environment. Alternatively, it could be driven by large growth and success.
The one thing that both scenarios have in common though, is the reliance on data.
According to Gartner, data has become one of the most valuable commodities available to organisations and increasing access to good quality information enables them to improve operational efficiencies, gain greater insight into performance, and ultimately improve decision making.
The issue that many face though is the lack of synchronisation between systems, which leads to the inconsistency of this data.
Organisations are regularly adopting new and different applications to manage day-to-day operations, whether this is for sales, accounting and marketing or development, production and distribution.
The breadth and scale of applications now available is quite mind-boggling; there seems to be an app for everything, and they are being widely adopted.
Whilst all these applications do make processing data for that particular role or department much easier, the applications often become isolated. This subsequently means that the data is also isolated, which can have serious implications for an organisation, as it can make it much harder for a company to gain insight into performance, business trends or opportunities.
These information silos therefore need to be synchronised to help improve business efficiency and streamline operational processes. As a result, all applications and systems within a business need to be integrated. Download the workbook below to help you establish in-house systems, data processes and day-to-day operations that require business process automation.
Some organisations may not necessarily be early adopters of lots of new applications, relying solely on legacy systems to run the business. However, the same analogy applies. And to be honest, it won’t be long before they will be adopting new technology. They will have to in order to stay competitive and to service the needs of their customers, partners or suppliers.
The beauty of system integration though is that it will help prolong the life of these legacy systems, many of which will have received significant investment over the years and likely hold a wealth of data.
Implementing system integration will enable organisations to add to their existing IT environment without ripping everything out. This is even more significant if an organisation acquires another business.
Quite often each organisation will have its own dedicated infrastructure. For example, one organisation could be using an ERP system like Microsoft Dynamics, but the company that has acquired it is using SAP Business One. Without system integration, this could create significant operational issues.
In a similar vein, having a system integration plan in place will also be much easier to incorporate and link up with partner systems if required.
This is especially relevant for many B2B businesses, especially within the manufacturing and wholesale industries, that need to exchange business critical documents and messages.
Even smaller businesses are now trading with the likes of Amazon, B&Q, Wayfair etc., and will therefore require some form of EDI (Electronic Data Interchange) integration.
System integration therefore makes organisations much more agile, with the ability to quickly adapt to new scenarios or opportunities.
- Multiple business systems and apps
- Prolongs life of legacy systems
- Adding a new application to the business
- Switching between systems and services
- Incorporate new partners or acquisitions
Going back to increasing efficiency, improving performance and cutting costs, organisations need to look at automating business processes, and this can only be achieved by integrating systems.
Manual data entry not only consumes valuable resources, it can also lead to inconsistent data and be susceptible to errors. Automating this type of activity will alleviate this issue and improve the synchronisation of data, leading to greater insight and analysis of information and the potential to pursue new opportunities.
Image: System integration enables organisations to integrate a wide range of business systems, databases and applications together
Challenges of System Integration
What is the biggest challenge in system integration?
System integration can either be very straightforward or extremely complicated. It really depends on the size of the company, the existing IT environment and the integration required. The common challenges of system integration may include:
Applications, systems and web services
One of the biggest challenges for a lot of older companies is that they are more than likely running operations using old monolithic, legacy systems. These have usually expanded over time, with different modules for specific tasks built in and interwoven into the fabric of the software. Often containing a huge amount of data (we’ll discuss data integration challenges later) organisations are, obviously, reluctant to replace them.
The complexity of these systems can often make them difficult to integrate with, especially if they are using old language formats or have been heavily customised.
In fact, it’s not that uncommon for businesses to have built a completely bespoke business system to manage day-to-day activities. This makes it even more challenging, especially if the developer is no longer working for the company.
Another challenge is the number of systems, applications and tools that are being used in the organisation. Again, these can widely differ on the type of connections required to integrate.
Some may even have existing integration connectors that need to be ripped out and replaced, adding further complications to an integration project.
Whereas system integration used to be fairly straightforward being contained on-premises within the enterprise, the widespread adoption of cloud-based applications and services has added another challenge to the mix.
Any integration projects therefore need to be able to cater for hybrid integration scenarios, with the ability to connect on-premises business systems with cloud applications.
In fact, the rapidly changing environment of cloud-based software means that system integration solutions need to be flexible enough to accommodate a wide range of applications and services. The long-term scalability of any system integration solution needs to be considered, as well as future project scenarios.
This is where some cheap, quick fix plug-ins fail. Although they can integrate certain applications quite quickly together, they are fairly limited in their scope and flexibility and often fall short of providing a more comprehensive service, usually only providing a basic connection.
They don’t usually have the capacity to transform data, access other data repositories or perform lookups. They tend to be quite rigid. Additionally, if anything breaks it can cause significant issues.
Ironically, many expensive, renowned solutions also suffer from this inflexibility, simply providing a fixed connection and lacking the customisation capabilities that many organisations require.
In rapidly changing environments, organisations need total control over design in order to adapt to business change, different requirements or new opportunities. The ability to modify an existing integration is therefore essential.
Another challenge that arises is the use of different messaging protocols and programming languages. As we have discussed, organisations usually have multiple systems, applications and web services that need to be connected. Each will utilise a particular integration method, whether this is via an API (SOAP or REST), EDI (electronic data interchange) or web hook.
Security is another factor that needs to be considered, especially in regard to the management of data and the concerns regarding data breaches.
An integration solution ideally needs to adhere to the best practices of the software vendor that it is connecting to, and ideally have some form of accreditation. It should also provide advanced authentication protocols, such as OpenID Connect, Azure Active Directory and OAuth2, to provide secure access for all users.
Data integration challenges
The whole point of system integration is to improve the management and synchronisation of data.
Data does, however, bring its own challenges, as you can imagine.
The quality of data is essential for any successful integration project. As we have discussed, organisations are using a multitude of different applications and systems which means that there are multiple data repositories. Collating all of this data suddenly becomes a massive undertaking.
Additionally, if an organisation hasn’t implemented a data integration plan before, a lot of these data repositories could contain duplicate information, or even the wrong information. Data will therefore need to be cleansed before it can become useful.
Sometimes the data may not even be located where it should be, causing further delays to integration. It could be saved in a spreadsheet on an employee’s desktop, hidden away in some dark corner of the database, or a multitude of different reasons.
Applications and systems can also generate different data formats which may not necessarily be compatible with each other. Some systems may write formats as XML, whereas the target system may require the format as a CSV (Command Separated Value) or JSON (JavaScript Object Notation) to read the data.
The data will therefore need to be converted into an understandable format for the receiving system or application to process. This is another tricky and time-consuming activity that not all integration solutions can manage.
When organisations are looking to integrate systems they also need to consider how the data is processed. Some solutions only process data one way. However, for data integration to have real impact they really need to consider bi-directional data flows. Download the brochure below to discover over 200 ways your organisation can benefit from business process automation.
Download 200 Ways to use automation
Projects and system integration planning
In light of all the above, one of the biggest challenges to system integration is ensuring that the planning is watertight.
Ineffectual planning is one the biggest downfalls in system integration projects. Far too many integration projects fail to deliver or don’t return an expected outcome due to a lack of research and forward planning.
Without proper planning and a quantifiable approach to system integration it could quite quickly unravel. A great deal needs to be considered before embarking on an integration project.
- What are you trying to achieve?
- What is the purpose of the integration?
- Which systems are essential to integrate?
- What data needs to be processed and managed?
Everyone, or at least each department, within the organisation needs to have some sort of input into the planning process to ensure that the project delivers what is actually needed.
In line with this, there also needs to be a level of accountability so that the project can run smoothly, whether this is the project manager or the actual systems integrator.
One issue that really does make system integration challenging is when an organisation requires the integration process to be immediate. Time constraints on projects can obviously be a massive challenge for integration projects – especially as it can reduce the planning time as discussed above.
An integration solution therefore needs to be extremely flexible, with the ability to quickly adapt to a changing environment and the demands of the project. Use the downloadable project management workbook below to help you detail your system integration requirements.
Download BPA Project Management Workbook
Types of data integration methods
As you may have gathered from reading this article, the method that you choose for system and data integration has a significant impact on performance, speed, security and flexibility.
Manual integration
Prior to the release of APIs, manual integration was the only real option. Developers would have to write bespoke code to connect two different systems or data sources together. They would then need to collect and cleanse the data and then convert it into the relevant format in order to upload it into the specified application or system.
This is an extremely time-consuming process, which takes up a lot of resource, and is prone to errors. Once written and applied it is extremely difficult to adapt or customise, which, as we have mentioned before, can be further compounded if the person who has written the code has stopped working for the company.
It is a very rigid and locked method of connecting data sources, and is only really recommended for handling a small amount of data that does not need constantly changing.
Application integration
The development of APIs, however, has made the integration process a lot simpler. Although still requiring developers to connect the systems, application integration, also known as point-to-point integration, uses a set of common API protocols to connect and transfer data automatically.
The APIs can be connected to each other by either in-house development or by using a simple plug-in. Whilst this is a vast improvement on manual integration, it still has its limitations.
Again, the integration tends to be quite rigid and doesn’t always provide the flexibility that many organisations require, such as the ability to customise the connection or data flows without significant disruption.
Scalability is therefore restricted with this method and constantly requires dedicated technical management onsite to ensure that it runs smoothly at all times.
The integration is also singular, simply deployed for one particular connection, making it only a small component of an often-required wider integration strategy.
Middleware integration tools
Middleware integration tools, on the other hand, is a much more flexible option for system and data integration. As the name suggests, middleware sits in the middle of an IT environment connecting all the various systems, databases and web services that an organisation uses, whether on-premises or in the cloud.
Modern middleware solutions also provide a much more intuitive way of connecting systems and adjusting workflows by using a visual, graphical user interface, with minimal reliance on coding. Data flows, as well as being automated, are easily adapted and reconfigured using drag and drop technology.
A competent middleware solution can also handle a multitude of data formats, with the ability to convert data from one source into a format that the receiving system can read. This makes it especially useful for connecting legacy systems with newer applications and services.
Although middleware may have a greater initial outlay, the ROI can often be quite instantaneous. It also provides a much more flexible and scalable option for future projects.
Electronic data interchange
Electronic data interchange (EDI) is another common integration approach that is widely used by many organisations, especially B2B businesses that need to share business information and documents in an electronic format, such as manufacturers, retailers, suppliers, carriers and 3PLs.
EDI is achieved by either a direct connection between the two businesses, or via a VAN (value-added network) maintained by a third-party.
EDI technology uses various protocols, including FTP/ SFTP (Secure File Transfer Protocol), HTTP/HTTPS-based protocols, mModem (asynchronous and synchronous), AS1/2/4 (Applicability Statement), SOAP etc., to exchange purchase orders, invoices, dispatch and delivery notes, debit and credit notes etc., in a standard format.
Despite the ability to connect directly, many organisations are actually now using a middleware solution to integrate other business applications with EDI systems, as well as to increase business process automation capability.
How do you implement system integration?
So what are system integration best practices? Well, the first action to undertake is to assemble a project management team.
Every organisation will have a different approach, depending on the size, structure and the knowledge of the employees responsible for the integration project.
Some organisations may have a dedicated IT department or person that is fully capable of managing the entire process. Others may rely on an IT partner or systems integrator to implement and guide them through the project.
Whichever scenario is applicable, the project management team will then need to embark on a 5-step plan of action, including:
- Planning
- Analysis
- Design
- Implementation
- Maintenance
System integration planning
As we discussed earlier in the article, system integration planning is extremely important, and could determine its success or failure.
From the outset, everyone needs to know the objectives of the project – what is trying to be achieved and the ultimate goal for system integration. This needs to tie in with an overall business automation strategy that will help improve efficiencies, increase performance and ultimately enable growth.
At this stage, it is advisable to get input from each and every employee or department that will be relying on the integration and subsequent data management to ensure that the right information is going to be available.
- What are the business cases – what needs resolving?
- What data needs to be accessed and managed?
- What processes need to be automated?
- What are the priority objectives?
- What are the future requirements?
The objectives will usually determine the type of integration that is required and the level of investment to be expected. It will also assist with prioritising the stages of the project. Use the downloadable project management workbook below to help you determine your system integration plan.
Download BPA Project Management Workbook
System integration analysis
Once the objectives have been agreed, the project management team then need to carry out a detailed analysis of the existing infrastructure – the systems, applications and processes already in place and where the data is located. This will help determine what integration method is required and what processes need to be automated.
- What systems need to be integrated?
- On-premises to on-premises?
- On-premises to cloud-based applications?
- A hybrid integration environment?
- Is integration with web services required?
- Are there existing integrations that need replacing?
- Which processes are going to be automated?
- How much data needs to be processed?
- What business rules are in place?
Once these questions have been assessed the systems integrator, IT partner or internal developer will be able to provide a detailed recommendation of what is achievable, in a particular timeframe, and an estimated cost.
System integration design
With the objectives agreed and a detailed analysis performed, the project management team can then select the right integration solution(s) to progress with the project.
This will provide the platform for the team to start designing the architecture and mapping the business processes.
A good way of achieving this is by documenting the desired business processes into BPMN (business process model and notation) process flow diagrams. These flow charts provide a visual representation of the actual processes that technical or non-technical people can understand.
These process flow charts can help map out administrative processes such as:
- Receiving orders from an eCommerce web store
- Expense or discount approvals
- Inbound customer support enquiries
- Placing orders with a supplier
- Distributing documents and reports
The process flow charts will also enable the team to ensure business rules are followed and help identify any potential issues or bottlenecks that might arise.
Image: Process flow diagrams can help map out the integration required and the processes that need automating
System integration implementation
Once the integration and processes are mapped out, the systems integrator or IT partner can then start building the connections and business process automation tasks.
The type of system integration selected will ultimately determine the length of time that this will take to complete. In many instances, manual integration will potentially take months due to the bespoke coding required. Using a middleware solution, however, can reduce this time to just a few weeks.
“If I had to start from inception, to deliver the intercompany solution bespoke would easily take me six to twelve months to deliver. We actually turned it around in probably four to five weeks. We’re still working on it today with further customisations but the core part of the project is 100% delivered. We’re managing it and we’re building and enhancing it. But that’s, without doubt, the quickest turnaround we’ve ever done. In fact, I’ve ever completed in my history of development.” David Kelly, Head of ERP Applications, Taoglas
Once the connections and tasks are in place the system will need to be vigorously tested before going live to ensure a smooth transition to automated processing.
Maintenance
One of the benefits of using a dedicated middleware solution over bespoke manual integration is the flexibility and scalability that is offered. New applications, web services or even new central business systems can easily be incorporated into an IT environment.
Data workflows can also be managed from one central location and easily adapted to react to any business changes.
Upgrades will also be applied without affecting any system connections or causing any interruptions to data flows. This would be relatively impossible with manual integration.
Support and training should also be available, enabling IT employees to manage the business processes internally.
Successful system integration projects
System integration and applying automation to numerous business processes, enables organisations to easily improve business efficiency, reduce costs and increase the visibility of critical information. Following the system integration best practices outlined above will ensure the smooth deployemnt of any integration project and help to achieve agreed business objectives.
For more information on the benefits of system integration and how it can help your business, download the brochure below or call us on +44(0) 330 99 88 700.
