Batch Processing in a Services World

BPEL and Job Schedulers are both relevant even as IT infrastructures migrate to real time services-based processing.    This article will explain how BPEL and job schedulers (most recently branded as Workload Automationsuites) provide an integrated solution that can satisfy the needs of batch and real time processing in a services-orientated infrastructure. Industry leading distributed job schedulers, workload automation (WLA) products, are offered from UC4, Orsyp, CISCO and Advanced Systems Concept, Inc. Oracle offers an industry leading BPEL Process Manager that runs on a variety of Java EE containers.

Full article here: http://bpelresource.com/wp-content/uploads/2010/07/BPELWorldArticleBatchProcessingInAServicesWorldv2.pdf

Figthing Process Fragmentation

Professionals all over the world in Information Technology are fighting the never ending battle against project creep, missed deadlines and cost overruns. The lack of success in doing so seems to indicate that there is a deeper problem that has to be solved first. After analyzing customer projects for 20 years, I may have discovered a key element of this problem. Well, it actually is not a unique discovery, because it is likely that every professional in IT has run into the same situation but has looked at the consequences and not at the cause.

It seems that process fragmentation is the root cause of most unsolved IT problems.

It starts with the meta-process of IT Change Management that requires that a business application (made up from processes, tasks and actvities itself) is first analysed, then developed, tested, integration tested, rolled out and then put into production by different IT departments that distance themselves ever more from the business user. Current Change Management has however emerged over many years because of a quality requirement that is totally unreasonable in its expecations and thus has driven IT applications off the cost scale. 99.99% availability makes sense for infrastructure but not for a business service front-end. It is also not necessary as we can see from Internet use.

Here a more human problem enters the landscape. What is it that management wants from IT? One of the interests is higher productivity, meaning that less people can achieve a certain amount of throughput. The second is ensuring the quality of the work performed independant of the people and ideally enable an untrained person to perform the work needed. People are in fact put last, and that creates the problem for IT. Putting people first – employees AND customers – would make a world of difference. People are actually seen seperated from the business when they really are the business.

The current approach to the above is to analyse the business process and encode decision making into rigid rules. The resultant simplistic 2D-flowcharts and IF/THEN rules can however not properly represent the business activity that the user needs to perform his job well and to user satisfaction. It is pretty obvious that a fragmented, rigid 2D flowchart cannot represent a 4D event-driven, dynamic world that is not fragmented. Process or application monitoring does not help, as it only tells you if the defined processes are executed as defined. Business intelligence might tell you that some expected numbers are wrong but not where to improve the process. Even if you know how to improve the process, you then need it developed, tested and put into production. This loop is long and expensive as mentioned before. The business also looses its ability to adapt to market changes.

Right here, IT Change Management has to change and consolidate with application or process development. Ideally, it would already include application or process analysis with the resultant documentation that becomes part of the application. Right here, it too becomes obvious that state-of-the-art application development using programming languages such as Cobol, Java or C++ with APIs are unable to cope. This is where the SOA concept developed that tries to create a flexible definable layer between the front-end application and the back-end service. But current SOA approaches do not deliver these aspects of Change Management and are built on either Java programming with UML modelling or jBPEL with BPM modelling. Extactly that creates another even more complex layer of fragmentation and spoils the potential benefits of SOA. Adding additional fragmentation layers such as outsourcing and governance simply does not seem the right approach to achieve shorter projects and more agility.

The application solution is to see business process not as step-by-step fragments but as a collection of business services that do not much more than bundle and hold the case related business communication and information content. The content is state/event driven and implicitly creates the progression of the business case to its completion. Business professionals must be able to interactively define the business services they need (I propose by recording or training) without the use of flowchart analysis tools that are completely abstract to a business user and do mostly require later use of programming tools anyway.

The current IT process segment of defining and testing such services (processes) must not be seen as a programming effort but as part of normal business activity. The business department must be agile enough to provide the input to the power users defining services and be willing to test and fine-tune such applications. A gradual and interactive development approach like that it not really new but was first suggested in 1990 as Extreme Programming using programming languages. The difficulty of achieving reasonable system stability with compiled languages ended that approach. The project benefits of Extreme Programming can however be achieved with an application platform that includes analysis tools, deployment and monitoring/tuning as part of it‘s Change Management.

In short, what IT needs is a defragmented approach to Change Management and a defragmented approach to creating business services (a.k.a. as processes). In fact, that implies that a much further reaching consolidation of user frontend processes is necessary, and that includes BPM, CRM , ECM and SOA.

Why a Business Architecture?

The main goal of a Business Architecture is to enable the business to improve customer service quality through a better transparency, flexibility and adaptability of business operations. The market environment changes more rapidly and the use of technology by customers dramatically influences how a business can operate. Financial services calculation processes, marketing programs, business rules and content change already weekly rather than monthly.

However, if a business architecture has to be modelled, encoded and assembled by using a large number of tools and software components it cannot provide the benefits. Today’s heavily fragmented and hardcoding-integrated IT systems (including SOA) are too rigid to enable rapidly changing business environments. Most IT departments do not focus on adaptability and innovation because they have been requested to focus on lowering cost and system stability. Therefore, six month rollout cycles are the norm with three month being the exception. Business users expectations of stability and executive demands for lower cost are incompatible with the ability to achieve a flexble and adaptive, competitive IT infrastructure. Efficiency is still the main IT goal, with effectiveness a far-off second and agility being no more than an overused buzzword.

Combine this with the misconception that running a business can be pre-planned and therefore encoded into processes and rules, with decisions being taken by predictive analysis based on historical (or better outdated?) business data. I propose that good business decisions are always taken by experienced people who use intuition to combine relevant data in business context. After billions of IT investments neither process management nor business intelligence have delivered the promised wonderland of the automated enterprise that the board can run remotely from the beach. Why?

Neither BPM nor BI consider the human side of running a business and therefore fail to produce a nimble, agile organization. Based on unproven management theories and over-optimistic information technology benefit claims a huge IT bureaucracy is now necessary to manage a complex technology stack. Control and use of the technology stack is only feasable through outsourcing partners and the necessary complex contracts reduce corporate agility even more. Billions are spent by the IT monopolists for marketing to sell an illusion of the IT-controlled business that does not exist and is not achievable by the proposed complex means.

The above situation was the reason for ISIS Papyrus to develop a new IT platform that does not require a huge technology stack and does not need complex programming but a simple modeling and rule definition methodology to build a flexible and adaptible Business Architecture that is mostly under the control of the business and not the IT department.

Agility AND innovation happen on the people level. BPM and SixSigma trash out the people empowerment slogan but fail to deliver because in neither approach people are given the freedom to do things as they see fit as long as the goals are achieved. Enterprise 2.0 is a countermovement to the bureaucratic IT-Governance approach, but if it is simply putting Web 2.0 behind the firewall without giving the user access to plausible business data entities there is not such thing as empowerment.

William of Ockham wrote in Numquam ponenda est pluralitas sine necessitate: “The explanation of any phenomenon should make as few assumptions as possible and not invent further entities to explain a theory.” He was a friar and felt that the one entity of God would explain everything. Bertrand Russel translated it to: “The simplest explanation is usually the best.” Translated further to IT means that coded software systems or process solutions that require substantial resources to be model a business and even more to then adapt it to changing needs make things more complex than necessary. Flexibility AND adaptability by the user – while ensuring transparency and maintainability – are the key capabilities of modern systems. SixSigma adds a lot of bureaucratic complexity that is certainly not in line with Occam’s Razor. Let’s simplify …

A detailed description of Business Architecture features of the Papyrus Platform you will on my Papyrus Architecture blog.

The BPM Delusion continues ..

I just read an invitation to ‘the most comprehensive, current and pragmatic BPM seminar in the world’. The host is claimed to be the most experienced and highly-rated seminar facilitator and author in the enterprise BPM field, who supposedly ‘developed more agile process-based business architectures and process models that will scale and adapt’ than anyone else. I won’t use a name but you can basically enter any BPM expert you want. They are all the same.

This wonders of BPM seminar should be attended by:

• Strategic Planners
• Business Process Executives, Stewards, Owners and Managers
• Business and Systems Architects, Analysts and Designers
• Business Executives and Managers
• Lean and Six Sigma leaders
• IT Leaders
• Program and Project Managers
• BPM Internal and External Consultants
• Change Agents who must influence cultural and behavioral transformation
• HR Professionals introducing new competencies and organization designs

Well, except for the executives and managers these people are not performing ANY work that produces ANY business revenue. They are all bureaucrats! There are now ten different people who will tell the ONE poor grunt sitting in the customer frontline how to do his job and once they come out of this seminar they will be brainwashed to believe that it will actually improve the way a business works.

The brainwashing starts with a certain amount of FUD (fear, uncertainty and doubt). The invitation points to economic pressures and change and an increasing burden of compliance that are the norm for any business. So what? Well, it says, ‘managers who solve problems on the spot employ short term thinking and create imperfect patchwork solutions.’ It claims that ‘only those with lean and reliable end-to-end processes will be able to do more with the scarce resources they already have’. Says who? Proves what? The worst short-term thinking perform c-level execs who look at quarterly results only to drive the shareprice up!

Then it says that ‘process management obviously needs to be aligned across the whole organization’. Does that not remind you of centrally planned communism? In any case they are telling you that you can’t manage your business and you can’t trust your managers and your people are idiots. They tell you that you do not need managers with initiative, experience, people management skills and who act decisive and intuitive. Nope, all wrong! You need bureaucrats who will put the business on track. Right. Exactly!

The invitation further proposes that the business does not need to know how to make a customer happy but ‘business managers, architects and analysts must be able to strategize, architect, define, understand, analyze, improve, and communicate knowledge about business processes for multiple purposes’. What are those purposes? The seminar will supposedly teach you how to manage the politics so that there won’t be any resistance. Basically it says right here two things: The people will hate it, so you need to enforce it and you do that by policing each and every step of the newly defined work processes by exploiting technology to the max! The Brave New World of 1984 is finally here.

It further says clearly that not people are the asset of the business but the processes are! Then there is a lot of mumbo-jumbo about scalable methods, models, best practices (a.k.a. copycat ideas) that can supposedly ‘be applied to whole enterprises’. Clearly that BPM expert has never tried that! Then the rest of the invitation is sprinkled with buzzwords about SOA-enabled, process-centric and model-driven, breakthrough BPMS technologies! Who believes this crap?

And what will you be able to do with all that new knowledge? Supposedly you will‘anticipate and respond to changing needs more quickly and deliver better performance faster’. It strikes me as strange that all this new bureaucracy will speed up change. Who will anticipate something? One of the ten BPM-Lean-SixSigma-ChangeAgent bureaucrats who are disconnected from daily operations? They think that the business clerks who are now paternally spoon-fed with process steps like idiots and who hate their BPM guts for taking away their initiative and customer orientation will tell them? That is the true DELUSION. This style of BPM motivation blatantly ignores the most basic, but most important concepts of human resource management.

Finally the course invitation puts the cards on the table by saying: ‘Get all people to change with less hassle’. Aha, the BPM pundits think that they can run the business by remote control. That is what they are selling. No manager in his right mind will believe that this is possible. Corporation-wide BPM is as bad as Outsourcing. It is proof of incompetence. It shows a focus on cost when there should be a focus on people – employees and customers!

My BPM bickering is often misinterpreted as being against process management. Well, the opposite is the case! But process management is about aligning people towards a common goal. That does not happen by nailing down every work step. A service business is not a manufacturing plant. Each customer is an individual and so is each employee. The quality is improved by better communication and monitoring, which is different to policing and enforcement! Employees are responsible for their own work and the quality they produce. Process management must help them to shape the processes how they need them. Managers and process management are ENABLERS. The huge change management overhead introduced by BPMS with all its tools, staff, centers of excellence and global process models will drag your agility down. Only when the bureaucracy is cut out and the change loop is short – which means WITHIN the process team – then a business can become agile.

I was told that there are not enough great employees so BPM it is necessary to improve quality. I absolutely do not agree, there are no bad people. They just can be in the wrong place and/or poorly managed. Well, guys – if your management gets BPM to help you, you know what they think of you. Finally, I was told that enterprisewide BPM and a great, motivated staff and street-wise firstline management are not mutually exclusive. Well, that is a dramatic delusion as well. You can only have one or the other. Great people go where they are needed and appreciated. So make your choice!

Latest SOA Need: Assistance with JBI Application Integration

The methodology of Enterprise Integration has advanced to Service Oriented Architecture (SOA) because of its ability to weave disparate applications and services to produce a business structure where data can flow as a business process. Environments have been built over time with diverse layers of applications. The development time and maintenance cost to manage these layers is driven down when integration components are built on standards. The most compelling standard in the integration space is Java Business Integration (JBI) which allows for the creation of a Service Oriented Architecture with interchangeable components that are vendor-independent.

Isn’t all development easier with a standard?

Not really. Don’t confuse the ease of using the standardized run-time components with the creation of those run-times. The creation of JBI run-time components is a whole new technology, with layered naming conventions and rules for each binding and service engine components. As with any new technology, JBI brings with it a sizable learning curve.

Plan time to learn JBI technology

Developers interested in JBI will need to reserve some time to learn. This type of exposure is mostly reviewing code snippets from existing open source JBI applications, like ServiceMix, and supplementing that learn-as-you-explore strategy with the 228 page JBI specification document publicly available from the Java Community Process organization. SOA integration experts, like Scott Ganyo with Moongate Technologies, agree that it can take a long time to learn the rules of the JBI spec nomenclature and nuances to code within the standard. While it is not brain surgery, even an experienced integration developer will need to devote targeted time to get up to speed.

The Holy Grail of Simplified Integration

So here is the hitch. Everybody is focused on SOA, and the push-point of that statement is everybody is everybody, from code crunchers to web designers.

The use of standards, like JBI, simplifies the combining of components but it is the ability to make those high-learning-curve standards accessible to all levels of coders that is the Holy Grail enterprises will be striving to acquire.

Less sophisticated audiences, one proven method

It is an early adopter assumption to think everyone using your product knows what they are doing. The first implementers of any new technology will cater to the most sophisticated users. But the SOA audience is as varied as the many applications they are trying to integrate, so this market will need a splay of products to cover different levels of expertise. A graphical interface, as we have seen successfully implemented in workflow and business process applications, is a proven method to simplify the design and implementation process. Just as BPEL maps out high-level business processes as a workflow, an intuitive graphical interface for the lower level integrations has a definite market for a technical user that wants to keep their distance from the detail specifications and simply drag and drop functionality.

Graphical Interfaces hide mundane details from developers

If vendors create graphical interfaces that generate standards-based code under the covers, a developer can avoid understanding the intimate details of the specification while still enjoying the benefits of developing a standards-based integration. A robust graphical interface allows each SOA developer to visualize the integration path and then simply click to define the properties in a fill-in-the-blank format. This expands the standards-based playing field to include a broader base of developers. The true beauty of the interface is to ultimately create standards based code so that the resulting run-time components integrate easily with other internally and externally developed components.

Graphical Interfaces are not always a priority

It is a certainty that more organizations will eventually go the graphical direction. First vendors develop the functionality and then they make it easier to reuse. While a couple of vendors have already started down a user-centric graphical path, an across-the-board improvement to robust graphical interfaces for integration will take a year or more.

Next Page »