abatishchev's blog

Once upon a time Le Petit Chaperon Rouge (LPCR) decided to change db_files from 200 to 300 on a busy cluster of four nodes.

Soon after the change applications started to block each other, timeout and crash (enq: TX - contention, cursor: pin S wait on X, enq: HW - contention). Those applications which were actively inserting into LOB columns - suffered the most.

Many lock holders were waiting for "DFS lock handle". Thus LPCR learned that DFS stands for "Distributed File System".

As soon as she reversed the changes - everything went back to normal.

Moral:

A handy alert log is invaluable for troubleshooting database problems. A RAC database has multiple alert logs.
I prefer to monitor them through a single table.

"A master failing to make an entry in the vessel's official
logbook as required by this part is liable to the Government for a
civil penalty of $200.

United States Code. Title 46 - Shipping. Subtitle II -Vessels and seamen, Part G - Merchant seamen protection and relief. Chapter 1113 -Official logbooks.

How meticulously do you keep your book log, Captain?
Luckily for us, our databases are as much ships as they are first mates. They are intelligent enough to keep their own records. And those logs are as important as vessel logs of the past, because the information carried by an early 21st century database could be easily worth more than 1450 tons of tea carried by Cutty Sark in 1870 en route from Shanghai to London.

As Oracle does all the mundane work, our role becomes more creative - to inspect and properly use the gathered information.