AAA selected five oil brands by first identifying brands commonly available throughout the United States. Only brands that produced both conventional and full synthetic oils (per the marketers) which met ILSAC GF-5 and API SN were considered. The SAE 5W-20 conventional and synthetic oil from each of the five selected brands was chosen because this viscosity is common in newer vehicles, and low viscosity oils will be increasingly popular in the future. Five quarts of the same lot number of each oil were purchased from auto part retailers in the greater Orlando area and sent to the testing labs unopened.

I think this is a pretty good sampling process, although it is possible that one or more of the oils was not current chemistry. I cant disagree with the viscosity grade or their rationale for choosing it, either.

When it comes to the tests selected, Ive got some questions. Well get to that in a minute. All the measurements specified by the selected ASTM tests were reported to AAA in unprocessed form. The tests were:

1. ASTM D6278 Kurt Orbahn Diesel Injector Test for shear stability, kinematic viscosity stability and average percent viscosity loss.

2. ASTM D4683 test for high temperature/high shear viscosity, HTHS stability and average percent viscosity loss.

3. ASTM D7097 Moderately High Temperature Piston Deposits by Thermo-Oxidation Engine Oil Simulation Test-TEOST MHT.

4. ASTM D6335 High Temperature Deposits by Thermo-Oxidation Engine Oil Simulation Test (TEOST 33C).

5. ASTM D5800 Noack volatility test (Procedure C).

6. ASTM D5133 Low Temperature, Low Shear Rate, Viscosity/Temperature Dependence by the Scanning Brookfield Test.

7. ASTM D4742 oxidation stability test by Thin-Film Oxygen Uptake (TFOUT).

8. ASTM D7528 Romaszewski Oil Bench Oxidation test (ROBO Apparatus).

9 and 10. ASTM D5293 Cold-Cranking Simulator (CCS) and ASTM D4684 Mini-Rotary Viscometer (MRV) tests for low-temperature pumpability after aging.

Heres where I have some concerns. None of the tests are actual engine tests but are, rather, bench tests. I dont have any problem with any of the tests (related to SN and GF-5), but they just dont capture the full impact of an automobile engine in actual operation. Each measures only a single parameter and none simulates the road very well.

Field tests are the truest measure of an oils performance, as Don Smolenski at Evonik Oil Additives (formerly with General Motors) noted in a Society of Tribologists and Lubrication Engineers webinar. However, it can be difficult to assess relative severity in field tests, and sometimes they lack good reproducibility. In addition, they are expensive, lengthy, and can be difficult to monitor and control. In an industry with rapidly changing requirements, field tests are not the primary means of assessing performance of engine oils.

Lets go to the opposite extreme-bench tests-which is where the AAA project went. Bench tests are inherently more reproducible. In addition, they are relatively quick and much less expensive, and even have consistent hardware. Sounds like a winner, doesnt it?

Not so fast. Bench tests have some pitfalls of which to be aware. First, its difficult to know if the test accurately reproduces any potential failure mechanism in actual field service. Second, bench tests often do not even remotely correlate with real world performance. Thats why OEMs are sometimes (maybe often times) not impressed by bench test results.

The answer lies somewhere in the middle, with engine tests. Engine tests can accurately simulate real world performance. They can assess relative severity through the use of reference oils and, if designed properly, have good reproducibility. The good news continues with the fact that engine tests are not as expensive as field tests and can often be completed in a few weeks or less. Engine tests are certainly more relevant than bench tests and are probably the best compromise available.

So if engine tests are the answer, how do they get developed? In his webinar, Smolenski notes that new tests are sometimes created in response to field issues; but most often the new tests are developed due to hardware obsolescence. There is also a need which has become more important recently, and that is to be proactive. Some new engine designs withhigher specific output may have high oil temperatures; OEMs need better oxidation stability of the oil. Other factors such as a more robust oil will allow longer drain intervals, etc.

If the new test is necessary because of test-engine obsolescence, the sponsoring OEM will try to estimate remaining life of the current test and when the new test must be ready. Typically they will notify the industry at least two to three years out. They will likely engage the appropriate ASTM surveillance panel as soon as they have a draft procedure to check out. As should be relatively apparent, a replacement test will probably have fewer issues of acceptance-its likely to be a must-have. A new test will see more scrutiny.

As a part of the process of comparing one oil against another in an engine test, and knowing whether or not a test accurately discriminates between oils that are good or bad, engine tests must also have reference oils. These reference oils can be good or bad, but ideally borderline. In order to select a reference oil, its necessary to get oils that are known to do something for (or to) engines. Generally, that means that reference oils are identified in field tests. Stated another way, field test oils are the most common reference oils. They cant be known by divine intervention! The source of reference oils is usually oil companies or OEMs. They can be superior performers and the source of the oil is quite happy to be the supplier. However, suppliers are loathe to be associated with poor reference oils, so these can be hard to get.

Reference oils, once they are selected, are managed by ASTMs Test Monitoring Center. The composition is confidential and the oil cannot be analyzed. That is necessary in order to maintain the integrity of the engine test system.

But what happens if two oils are run through the engine sequence test protocols for say, API SN-Resource Conserving? And what if one oil provides better numbers (47 percent) than the other oil? Is one oil superior to the other? Surprisingly enough, this question has come up in several instances and the answer is, not necessarily.

In one famous case (Castrol vs. Pennzoil), federal courts determined that you cannot claim superiority based on extended length tests. The 2000 case involved comparing double-length Sequence IIIE results on competitive oils and making a claim that, because one oil gave better results, it was superior. The court said not so, because a double-length test was not standard and therefore lacked validity. Ouch! To state it plainly, passing an engine test is sufficient to make a claim of quality and relative differences in test results are not going to be meaningful. The answer to testing for superiority claims is field tests. So we come full circle.

The point to make here is that comparisons of oils, even in engine tests, can be very touchy. To use bench tests for comparison, as AAA did, is not truly representative. There are too many factors at work to make a claim of success based on viscosity, a volatility test or a shear stability test. These tests do demonstrate a specific property and how oils compare by that property, but they arent a true measure of the total oils performance.

Certainly, a comparison between a synthetic and a conventional oil is valid insofar as the base oil is concerned. I also have to ask if the additive package used in the samples run by AAA was the same for the synthetic vs. the conventional oil for each company. That would eliminate the additive from the equation. There is also the question of which synthetic was used.

As for consumer demand for synthetics, it is likely to increase given that they are available at more competitive prices coupled with the increased cost of automobile ownership for consumers. National Oil & Lube News found in their 2016 survey of fast oil change outlets that 24 percent of customers were getting synthetics in their oil change even at double the cost. Oil change intervals were around 5,000 miles, and vehicles being serviced were on average nine years old with 90,000 miles on the odometer.

The bottom line is that the AAA report is a very interesting exercise and sheds a lot of light on quality issues related to engine oils. The results of the tests definitely show advantages for synthetic over conventional, but Im not comfortable with assigning a percent improvement value for synthetic based solely on bench tests.

The next step should be a comparison of engine test results, but that will take big bucks. Nevertheless, a tip of my hat to AAA for taking on this complex subject and creating much needed discussion.

Industry consultant Steve Swedberg has over 40 years experience in lubricants, most notably with Pennzoil and Chevron Oronite. He is a longtime member of the American Chemical Society and SAE International, where he was chairman of Technical Committee 1 on automotive engine oils. He can be reached at steveswedberg@cox.net.