One of our Mailing List Subscribers wanted to know how we would go about testing a product and determine whether it was suitable for use.

I described the testing method we use below using a product that for the purpose of this document we will call Product A. This product is tested under this scenario to determine whether it has a beneficial effect on corals.

Below is my answer :



As per Dr Sanjay's request below is the testing method that would be used
for a hypothetical product we will call  "A".

Pls read the entire message before jumping to conclusions. Thank you.

This is meant to describe the procedures used in a general case and is not
the only one as, depending on what one is trying to accomplish, variations
on the methods used may have to be added or introduced.


-Acquire 10 log books for notation of observations on each tank
-Set up 10 separate 15 gallon aquariums
-Water all treated and measured for microsiemens rating before any salt is
added to ensure it is of the same quality
-Same salt and same water source used in all tanks
-Measurements in tank done before adding any corals to adjust parameters to
reflect generally accepted levels of pH, salinity, iodine, calcium etc...
This may require some time to balance the tanks out to equalize at
approximately all the same values.
-While this testing and adjustments are done make sure the systems are not
cycling.
-Use identical lights over all tanks. Strength depending on type of coral
that is present in the aquariums and what type of lighting in photon
irradiance it requires. Use identical photoperiods.
-Place tanks in temp controlled room so tank temp does not vary
-keep tanks out of any natural light
-Stock the aquariums with the same coral of approximate same size, obtained
from same supplier
-Recheck paramaters after 3days to ensure all are within the normally
suggested parameters and close enough to each other.
-Same skimmer used on all tanks with same airpump and same airstones or in
the event small venturi ones are used make sure they are all identical
-Number tanks from 1 to 10
-Let tanks age for a few weeks (4-8) so they stabilize and take water
quality parameter measurements. Timeframe depends on aquariums. Before
adding any coral to any tank we wish to ensure that tank conditions are
nearly identical
-no substrate
-no live rock
-no fish
-only 1 coral per tank
-After stabilization period has passed and tanks are judged to be close to
each other in all parameters coral is added to each tank
-Measurements taken twice  a day of water conditions and adjustments made
if necessary as time goes on. Main tank parameters kept as close to each
other as possible (includes pH, salinity, temp, O2 and so on)
-if any additives other than A are added identical dosages are dispensed to
each tank
-all and each action taken on a tank is logged in its log book
-when tanks are considered to be sufficiently aged and are showing near
identical water quality parameters the test can begin

The same method is now applied to another set of 10 tanks in which a
different coral will be placed

The same method is applied to another set of 10 aquariums in which a fish
will be placed.

All in all the procedure requires 30 15 gallon tanks.

(Note that at one time while we were testing we had over 60 of them in a
special room at our Bridgeport location in which nothing else was done than
running the tests)

The set up before the start of the testing is now concluded.

In each batch 5 tanks are selected that will serve as the test tanks and
the other five will be tanks in which no testing occurs but which will be
used for comparison reasons.

Individual X determines which tanks will be used and logs this in a
separate logbook to which only he or she has access (in our case it was a he)

Individual Y is now provided with 3 sets of ten vials labeled from one
through 30

Out of each batch of 10 vials only 5 contain the actual supplement and the
other five contain plain water or a totally neutral additive that has no
effect on the corals or fish.

The vials are filled each day from fresh batches prepared by Individual X.
Individual Y only dispenses them but has no idea which vial contains the
product A and which one does not. All individual Y does is dispense.

Daily records are now kept of what ever observation individual Y and Z
notice on the aquariums. Individual X does not make any notes on the
aquariums in the various log books.

Testing is done on all tanks to ensure that they remain within the
recommended ranges for the parameters that are controlled (pH temp etc).
Any change made to any tank is recorded in its respective log book

The amount of product A added is dosed in such a manner that a once a day
addition is all that is dosed to each aquarium (in reality only 15 of those
contain product A)

Vials are numbered to correspond to the tank numbers so that there can be
no mistake and that a vial that contains product A accidently ends up in a
tank that is not supposed to receive this additive.

Thus:
Individual X prepares the vials
Individual Y dispenses them
Individual Y and Z make notations in the logbooks at to what their
observations are in terms of how the corals look (expansion, growth, and
anything else they may find siginificant that is different from what the
animals looked like the day before).

Every day a photo is taken of each tank at the same time of the day. These
photos are labeled by date and by aquarium number (to facilitate things and
prevent errors the  tanks are actually numbered with a marker so that when
a photo is taken the # of the tank appears in the photo).

The test continues for approximately 4-6 months and records are kept daily
and so are photos.

Individuals are told that no changes can be made other than the ones agreed
to indvance and that are considered necessary to validate the testing (this
applies mainly to pH, temp, salinity, etc).

After the test period is concluded a review of the records takes place and
photo comparison starts. The first photo in each batch is one of the animal
on the day before product A is actually added to 15 of the tanks.

Sets of all photos are copied and a total of six sets made.

A complete set is sent to individual who are not associated with the company.

They are requested to evaluate what they see over time and have no access
to the logs.

They are asked to select 3 photos out of each batch and label those as
best, second best and third best.

These photos are returned to individual V.

Individuals V Y and Z now get together and review the input of the
outsiders and determine in each batch what time of the experiment these
photos actually correspond to.

These photos are compared to the originals of the tested specimens.

If the photo that represent the animal before the testing started appears
in any of the selections the test is determined to be negative and the idea
is abandoned.

If the orginal photo does not appear in any of the batches that are
returned as the selection the test is considered to be positive as it would
appear from those selections that based on an independent review the
animals looked better after product A was being added.

Now comes the phase of determining what timeframe these photos correspond to.

If any of the photos returned as part of the best etc selection falls
within the first 10 days of testing the test is determined as inconclusive

If none are from within the first ten days of testing the test is rated
conclusive.

Now the dates of each photo are determined based on the markings made on
them that are not the actual dates but a manner of refering to the date
that cannot be identified by the outsiders.

For the test to be really conclusive the 3 rated photos in each batch
(coral 1, coral 2 and fish 1) should ideally be within a time frame that is
close. 

If this is the case the test is rated as conclusive.

If on the other hand the photos selected as best etc. differ too much in
terms of what stage of the experiment they refer to, the test is again
decided on as being inconclusive.

if the photo selection time framses of all reviewers are close to each
other the experiment is determined to be conclusive and consideration will
be given to run market the product.

If the selections are inconclusive but somewhat close together the whole
testing may be started over again to see if different resutls are arrived at.

If the test was rated conclusive and a decision is made to possibly market
product A, a new batch of 10 tanks is set up with a different animal yet
(this time only 10 tanks are used and again only 5 will receive the
additive in the same quantity.

The test is now ran for the amount of time that corresponded to the time
frame that the photo selections corresponded to + 2 weeks.

Photos are taken again (basically the same procedure is followed as desribed)

These photos are now sent to 3 different individuals for best second best
and third best ratings again.

The photos chosen by these individuals should correspond to about the same
selection as the one made by the first objective reviewers

If this is the case, the experiment is determined to be positive and the
product is marketed or the change in a product is implemented.

I may have left some smaller steps out but these are pretty much the steps
that were followed for testing on whatever we tested. Some detail may have
been left out but the method described is pretty much a routine after a
while and not likely to run into errors.

The outside individuals have no idea what product is tested and what
results are being looked for of course. All they are requested to do is
evaluate photos. In most cases a script of questions will go with the
photos to assist them in their evaluations.

A last remark perhaps, the logbooks are controlled by individual X each day
to ensure that nothing was done to the aquariums that may skew the tests
and that only changes were made that were considered as not affecting the
testing.

If for some reason (and it happens) an animal dies or becomes diseased, the
tests are continued without that tank being included in the batch

If on the other hand, disease or stress is evident in a number of tanks
either product A is not suitable of the concentration or dosage used needs
to be reviewed and changed and the whole experiment needs to be restarted

All equipment used on each tank is propriatory to that tank and is not used
on any other tank.

After testing all aquariums and all equipment are sterilized.

The log books are destroyed after the experiment has been completed and
only summaries of each log book are kept for company use and justification
of the expenses incured should it be questioned by the IRS (and we were
audited on R&D once).

All photographs are kept for 18 months and then disposed of.

Since in the case of product A is is not know what we are trying to
accomplish it is difficult to put together a review sheet for what
characteristics to look for when evaluating the photos.

As you may have realized what is important is that the original specimen is
not part of the photo selection, that no photo taken within the first ten
days is part of the selection and that it can be repeated over more than
one testing sequence.

it is also important that the selections correspond to about the same
timeframe as that enables refining of the dosages used after it has been
determined that they are beneficial.
Perhaps we can now ask other manufacturers to disclose their testing
methods and see what their responses are.

Note that once one has all the equipment the cost of running other test is
not exhorbitant. it is acquiring all the necessary equipment the first time
that involves an sizeable investement.

Note that photos are taken in such a manner that the animals tested are
always taken from the same angle and by someone experienced in photography.

In one case we took bracketed photos of a test we conducted and ended up
with literally thousand of photos. Fortunately when one deals with this
kind of numbers the cost per photo goes down considerably.

You will probably wonder what happens to the animals used in the testing
(30 in most cases). They are donated to schools, aquarium societies or
public aquariums or are sold back to one or several stores.

Acquiring the stock is a sizeable expense in the testing.

It goes without saying that the photos choosen need to be from aquariums
treated with product A (in this hypothetical case).

I thought I would leave that as the last remark.

Note that some experiments are conducted to determine whether positive
results are obtained whereas others are conducted to determine what
negative effects ensue from the use of a certain product.

The type of test conducted affects the instructions the independent
reviewers receive.

Do all tests lasts this long. No, it depends on what is being tested. For
example when I first found a way to eliminate phosphate from aquarium water
(yes I did introduce those compounds to the hobby), the test did not
require outside reviewers at all. In such a case different batches of water
are loaded up with nutrients and the product is then used in various
amounts to determine whether or not it lowers these nutrients and whether
this can be repeated and repeated over different batches of water using
slightly different pH levels, salinities etc.

In addition tests are conducted to deternine whether the compound affects
anything else in the aquarium. Such tests require no photos and no animals
at first, but require a large number of different test kits.

once the fact is established that the product does remove a nutrient (e.g
PO4) the tests are then conducted with small quantities with at least 4 or
5 animals in the tank to determine whether the animals are impacted at all
by the presence of the compound or by the removal of the nutrient.

Many variations need to be used depending on what exactly one is trying to
accomplish and we have done so.

I hope this gives you a better view of our testing methods and will
alleviate your concerns that products are not properly tested or not tested
at all (it would be criminal I think to market such a product).

Note that this procedure was used for products brought to us by hobbyists
as well and that I can assure you many of the ones hobbyists offer and want
to market under the Thiel name for a sizeable royalty never got further
than maybe one or two weeks of testing because it became obvious very
quickly that the product was not suitable.





Albert