Golden Gate abends due to zeroed out records in trail file...

If you are getting odd abends on your Golden Gate replicats due to trail file corruption, make sure you visit your filesystem options for the filesystem where the manager process writes the trail files.

Long story short is that we were finding large swaths of Zeroes appearing in our trail files. Using logdump I could see the header of the trail file was OK, but somewhere down the line a valid record would be followed by straight up zeroes. Lots of them. I verified this using logdump and a hex editor.

Only by comparing the trail files on the source vs the trail files that made it to the target could we determine that the Extract was working correctly... but something was going awry by the time the file was stored on the target filesystem.

After updating our NFS mount options to match Oracle GoldenGate Best Practice: NFS Mount options for use with GoldenGate (Doc ID 1232303.1) everything settled down.