|
The intent of this page is to provide a forum to address various points of misinformation concerning PLOG processing, and thereby provide the reader with a better understanding of the issues involved with ADABAS Protection Log (PLOG) handling.
The following topics are presented and discussed:
If you have additional questions or issues that you would like to see addressed within this forum, please feel free to contact us.
The Challenges in Handling PLOG Data (a.k.a. “The PLOG PROBLEM”)
There are five basic challenges to handling PLOG data:
- The data on a PLOG is in compressed form.
Solution: This issue can be resolved by using the ADABAS ADASEL or ADACDC utility (mainframes) or ADAPLP (UNIX & Windows), all of which efficiently handle the decompression of PLOG records.
- A PLOG file contains transactions which occur against all files in a given ADABAS database.
Solution: This issue can be solved by using NatCDC, which will select records of interest for a specific file from the output of either the ADASEL, ADACDC or ADAPLP utilities.
- The transactional records in a PLOG are identified as either a “Before” record image or an “After” record image, and these “Before” and “After” record images must be analyzed to derive the type of transaction involved (i.e. a “Store”, an “Update”, or a “Delete”).
Solution: One solution to this issue is by using the ADACDC utility, which will distill transactions down into a Delta, and will additionally allow for the deliver of only those transactions that were actually committed.
Alternatively, there are some very simple rules that govern the interpretation of matching “Before” and “After” images to derive a Store, Update or Delete. Once these rules of interpretation are understood (this interpretation is built-into NatCDC), the proper handling of PLOG transactions into specific transaction types is easy, and therefore the output of ADASEL can be used.
In regards to the rules involved with processing ADASEL or ADAPLP output into the proper transaction type (ADACDC provides this functionality already), it should first be understood that all PLOG records contain a “transaction header”. This transaction header contains things like the file number of the affected file, the unique Internal Sequence Number (ISN) of the transaction record in the specified file, a Timestamp of when the transaction took place, and a flag which indicates whether the record is a “Before” or “After” Image type.
When an Update occurs in ADABAS, this Update will be reflected in the PLOG with both a “Before” Image record and an “After” record image. When a Store occurs, this Store will be reflected in a PLOG by a single “After” Image record. When a Delete occurs, this Delete is reflected by a single “Before” Image record.
To reconcile an Update transaction, a Before image and an After image will have comparable Timestamps along with the ISN. If any given Before or After image against the same ISN cannot be matched, then based on whether the image type is a Before or After, then the transaction is either a Delete or a Store.
This is straightforward data processing, and this logic can be hand-coded. Alternatively you can use the generational capabilities of NatQuery / NatCDC to handle this for you.
- PLOG records can be variable length.
Solution: The “trick” to handling these variable length records is the understanding of how recurring fields are physically delivered from a PLOG after decompression is performed. For every recurring field in a decompressed record, these recurring fields will be preceded by a special “occurrence count” (“C*”; or “C”-Star) field, a field that contains the number of occurrences of that recurring field. By reading the variable length record in relation to the File Description Table (FDT) that describes the target file - the location of these “occurrence counts” can be programmatically deduced, and then the following number of physical occurrences of data can be properly parsed.
The core NatCDC module handles the interpretation of PLOG records, and delivers fixed-length output records from variable-length PLOG input records.
- PLOG records can contain data that requires translation, such as Date and Timestamp fields.
Solution: For every data type that ADABAS can contain, there exists conversion routines that will take these data types and convert them into usable equivalents. Since Natural is particularly adept at addressing these data conversions, it is “natural” for Natural to process the PLOG output provided by ADASEL, ADACDC or ADAPLP - or you can use NatCDC to generate the required program(s) for you.
To summarize, the challenges of PLOG data are not really any different than the basic issues presented in general data processing - and these issues CAN BE completely handled by in house programmers. Alternatively, a Software AG customer can utilize tools such as NatQuery and NatCDC to automatically deliver the required processing.
[Back-To-Top]
No Other Product Can Handle Changes In ADABAS
In addition to the NatWorks’ product NatCDC, there are at least two other products on the market today that can address proper PLOG handling, not to mention the fact that customers can even write these processes themselves through the use of the ADABAS utilities ADASEL, ADACDC or ADAPLP (all of which are provided free with ADABAS) and Natural. The simple fact of the matter is that there are alternatives available, and in proof of this, we offer the following links to two separate companies (besides NatWorks) which offer PLOG handling solutions:
Our partner;
At NatWorks, we specifically invite you to check out these alternatives, and compare and contrast the combination of NatQuery and NatCDC to them. After this comparison, we are confident that you will see the power, flexibility and cost-effectiveness of the NatQuery / NatCDC solution. For an idea of the performance characteristics of the NatCDC solution, please visit the PLOG Processing Information page.
[Back-To-Top]
Using ADASEL is a “severe handicap” and is an “inefficient use of CPU”
There are indeed challenges to overcome when handling PLOG data, and these topics are discussed in the above section entitled The Challenges in Handling PLOG Data
Of the five major challenges involved in handling PLOG data, the decompression challenge is the most significant. It is the decompression algorithms that cause the most CPU overhead when handling PLOG data - and there is ABSOLUTELY NO PRODUCT on the market today that will decompress PLOG data faster or more efficiently than ADASEL.
We offer the following findings presented by Software AG customers on the SAG-L List Server (this thread can be found in the SAG-L archives).
It must be remembered that PLOG processing is a constantly recurring activity on a system, and excessive CPU usage will have an impact on overall operation.
Rather than being a “severe handicap” or “inefficient use of CPU”, it is the intelligent use of PLOG utilities that offers incomparable performance and value over any third-party product that attempts to handle decompression on their own. By utilizing the ADASEL, ADACDC or ADAPLP utilities as an integral part of PLOG processing, the NatWorks tool, NatCDC, leverages what you already own and automatically builds processes that are routinely used.
[Back-To-Top]
Backed-Out Transactions Are A Problem
A “Backed-Out” transaction is a transaction that has been physically applied to the ADABAS database, but for whatever reason this transaction is subsequently reversed (removed) from ADABAS. There are several reasons why this might occur, however in most applications, this situation will be a relatively rare occurrence.
To understand why Backed-Out transactions occurring within an ADABAS PLOG is a concern but not necessarily a problem, one must first understand that ADABAS uses an “optimistic update” approach to transaction handling. This “optimistic update” approach means that when a Store, Update or Delete transaction occurs against ADABAS, ADABAS immediately applies these actions to the database - even though the physical command to make these transactions “stick” (I.E. an End Transaction command or “ET”) has not yet occurred. In this interim period, ADABAS will show this new data to all users if it is requested, even though it has NOT been “ETed”.
Prior to an End Transaction being issued, a Backout Transaction can occur either programatically by issuing a Backout Transaction or “BT” command (this would be a rather bad programming practice however), or this can automatically occur by ADABAS itself when it senses that too much time has elapsed since the Store, Update or Delete was issued with no “ET” or “BT” having been subsequently issued (this is referred to as a “Transaction Timeout”).
If a Backout occurs in ADABAS, whether programatically or automatically by ADABAS itself, the Backout will be reflected in the PLOG by the creation of Record Images that precisely reverse the previous transaction(s) that previously occurred. So: As long as whatever product / process that is being used to handle PLOG transactions properly processes the contents of a PLOG, these backouts will be properly handled and interpreted.
[Back-To-Top]
Transactions That Span PLOGS Are A Problem
As ADABAS operates (and assuming that Disk Logging is used as opposed to logging to Tape), ADABAS typically writes transactions that occur against ADABAS to one of two dedicated disk files that contain PLOG dataset. When one PLOG disk file is completely filled, ADABAS automatically switches to writing transactional data to the second PLOG dataset, and at the same time initiates a process that copies the contents of the first PLOG dataset to tape or disc. In this manner, ADABAS should always be able to write transactional changes, and previously written transactional changes can be appropriately archived.
The “problem” referred to above has two aspects:
- ADABAS applies a transaction to its database that is recorded on one PLOG dataset, then a PLOG switch occurs prior to an End Transaction (ET) or Backout Transaction (BT) for that transaction, with the ET / BT physically showing up on the next PLOG dataset.
- An ADABAS Update transaction is stored into the PLOG as both a Before image and an After image. This somehow gets “split” between two PLOG datasets when a PLOG switch occurs in the middle, leaving a “Before” image on the first PLOG dataset, and an “After” image on the subsequent dataset. This concern would never apply to either Delete or Store transactions - as these are always shown in the PLOG as a single record flagged respectively as a “Before” or “After” image.
In regards to the first aspect of this “problem”, the answer lies in understanding how ADABAS’ “optimistic update” works. This then is the same issue as described in the Backed-Out Transactions section, and if your tools have the capacity, is not really a problem.
In regards to the second aspect of this “problem”, per Software AG log# 124049 and in reference to ADABAS version 6.2.2, an Update transaction will NEVER be split across two PLOGS. Subsequent to providing NatWorks with this information, Software AG subsequently reversed themselves on this information and then indicated that a BI from an Update transaction COULD be written to one PLOG dataset while the Update’s corresponding AI could be written to the next PLOG dataset.
Given that Software AG now indicates that a split could occur, the ability to properly handle these split transactions can be accomplished in one of two ways:
- If the intent of PLOG processing is to apply the changes made in a production database to a target once a day, then PLOG processing will most likely occur when online processing is essentially ended. At that point - if a PLOG switch is performed and all outstanding PLOG datasets are then concatenated together - there will be no “split” transactions.
- If the site wants to process PLOGs at multiple times throughout a given day - then the site may wish to use the ADACDC utility which addresses “hanging” transactions (and holds them appropriately).
[Back-To-Top]
Transaction Referential Integrity Is A problem
This issue is in essence the same “problem” as described in Backed-Out Transactions Are A Problem.
The crux of this issue is that ADABAS uses an “optimistic approach” (transactions are made available systemwide immediately once they are made and prior to them being ET’ed or BT’ed) to transaction handling, and virtually all relational databases use a “pessimistic approach” (meaning that transactions are not made available systemwide until after the transaction is fully finalized).
Regardless of approach, there will always be situations in which the referential integrity of specific records in a target warehouse are compromised at the point that a PLOG switch occurs, and it is currently unworkable (for NatWorks or any other PLOG vendor) to attempt to guarantee “referential integrity” on a relational database that uses a “pessimistic approach” to transaction handling when data is coming from ADABAS’ “optimistic approach”.
In light of this, we at NatWorks feel that an ADABAS Change Data Capture solution for ADABAS should embrace ADABAS’ “optimistic approach”, an approach that provides data that exactly reflects the status of an ADABAS database at the particular moment that a PLOG switch occurs. This approach then results in a “True” point-in-time of ADABAS.
[Back-To-Top]
What if I Only Want “Committed” Transactions in my Warehouse?
If it is a requirement that the target Data Warehouse / Data Mart should only be populated with ADABAS transactions that were committed (versus populating the target Data Warehouse / Data Mart with “optimistic” transactions that may not have been committed): Software AG provides the necessary tool to do this in the ADABAS utility, ADACDC.
The ADACDC utility processes PLOGs in a similar fashion to ADASEL (identifying transactions of interest and then decompressing these transactions), however ADACDC will “hold” transactions that have not yet been ET’ed whereas ADASEL makes no attempt to hold them. ADACDC then processes those transactions that were “held” against the next PLOG dataset to be processed, and then matches these “held” transactions to their respective commits.
Another difference between ADACDC and ADASEL is that ADACDC will automatically produce a “Delta” of transactions for you - whereas when utilizing ADASEL this “Delta” must be programatically handled (optional NatCDC process).
If processing only “committed” transactions into your warehouse is a requirement, then NatCDC can handle the output of ADACDC as easily as the output of ADASEL. NatCDC will provide the required processing to convert variable-length records to fixed length format, drop any unneeded fields, and convert ADABAS data types into ASCII equivalents, however there is no requirement for a Sort step or to deduce a “Delta”, when using the ADACDC utility.
[Back-To-Top]
[Back to the NatCDC Information Page]
|
