9.

Feature on Benchmark parameters Form

Next script wait: This is the time difference between two consecutive users

Save timing data:  This flag controls whether detailed timing data is collected. The Benchmark tool can be used to create large databases where the timing data is not relevant and will only fill up space.

Prioritized computers: A more correct term would be weighted computers. The Benchmark Users are dispensed according to the computer weights.  This is performed based on the available computers list, which is only updated by pinging.

Immortal: When executing Benchmark runs, the entire run will fail if one user fails. Therefore the Benchmark tool will automatically stop the execution in situations where a user fails. The Benchmark tool can be used for durability or stability testing. During such a test, it is preferred that the tool does not stop just because one client receives an error. The tool can therefore be set to run immortal, so that regardless of one client failure, the test will continue.

Step distribution: When running display-independent, the execution can be difficult to follow. For this reason, each user will report their progress step by step.  When a run is in progress, a button on the cockpit can provide information on the distribution. (See Figure 12: Step distribution)

@%20=58725AB5=.files/image012.jpg">

Figure 12: Step distribution

A Benchmark test can be started by clicking start run on the cockpit. This will create the user-scripts that are directions to each user participating in the run. Each user is assigned a script according to their computer’s weight. If the prioritized computers are enabled, each Benchmark user is assigned a specific computer to run on. The benchmark status is changed to Start run, with a specification on which run to start.

After a few seconds, the current run information in the cockpit should be updated to reflect the started run. When a run has started, it is possible to stop it by clicking Terminate run. The current step distribution can be activated by clicking on the same button in the cockpit.

Figure 12: Step distribution on the previous page shows an example of the step distribution output. It shows 10 users executing in the Sales order script (SO-BS v2-5). 1 user is currently creating a sales order, 2 users are creating a new line, and so forth. 23 seconds has elapsed since the run started. 22 lines have been created. These figures provide an estimated throughput of 3443 lines per hour. The throughput is calculated more accurately once the run is finished.

Run status

The different run statuses are:

•Withhold

–The run has not yet been started

•Waiting-–The run has been started, but not the user scripts.

•Started-–See step distribution for a progress (activated in Parameters)

•Error-–If a user receives an error, the entire run is terminated (unless Immortal)

•Terminated- the run has been terminated unsuccessfully.

•Finished-–All runs have been successfully completed. This state can only be reached if the Cockpit is open.

The cockpit should be kept open during the run. Every 30 seconds it will recalculate the status of all Benchmark users. If one user encounters an error, the entire run will be terminated unless the immortal flag is set (See Figure 11: Benchmark parameters). When all users are finished executing their user scripts, the cockpit will finish the entire run before calculating the statistics.

The laboratory concept

The Benchmark tool can create a large amount of data within a short period of time, however, the data created might influence the results. Therefore, the data collected by the tool can be exported and imported into a central database for analysis. In order to assure that all data can be imported in the same database and company, the concept of a benchmark laboratory is introduced.

Every Id for a Benchmark run consists of the following:

<Lab Id> <Lab serial number><run serial number>

Laboratory issues are handled in the form Laboratory Registry (See Figure 13: Laboratory registry) This form consists of a Laboratory and a number of series. When a new Benchmark company is created, a laboratory with the name ‘Test’ is created together with its first series.  The first Benchmark run will be called Test0000010001. All test runs should be performed in the test lab. Whenever tests are created that should be saved for later use, the user should create his/her own Laboratory.  At Navision we use the Lab Id to distinguish measurements on IBM, Compaq and HP computers. When the record-breaking test with Compaq was performed in Nice in June 2001, the Lab Id “CPQN” was assigned to all runs. Whenever the database was restored or the course of the tests was changed, the serial number was increased.

@%20=58725AB5=.files/image013.jpg">

Figure 13: Laboratory registry

To partition the runs into groups, the serial number should be used. The statistic results, along with the computer setup associated with a given Laboratory, can be exported and then imported into a central database for analysis. The laboratory registry in Figure 13: Laboratory registry is a snapshot of a central database at Navision.  Results have been imported from various locations so that they can be analyzed across platforms.

Feature on Benchmark parameters Form

Next script wait: This is the time difference between two consecutive users

Save timing data:  This flag controls whether detailed timing data is collected. The Benchmark tool can be used to create large databases where the timing data is not relevant and will only fill up space.

Prioritized computers: A more correct term would be weighted computers. The Benchmark Users are dispensed according to the computer weights.  This is performed based on the available computers list, which is only updated by pinging.

Immortal: When executing Benchmark runs, the entire run will fail if one user fails. Therefore the Benchmark tool will automatically stop the execution in situations where a user fails. The Benchmark tool can be used for durability or stability testing. During such a test, it is preferred that the tool does not stop just because one client receives an error. The tool can therefore be set to run immortal, so that regardless of one client failure, the test will continue.

Step distribution: When running display-independent, the execution can be difficult to follow. For this reason, each user will report their progress step by step.  When a run is in progress, a button on the cockpit can provide information on the distribution. (See Figure 12: Step distribution)

@%20=58725AB5=.files/image012.jpg">

Figure 12: Step distribution

A Benchmark test can be started by clicking start run on the cockpit. This will create the user-scripts that are directions to each user participating in the run. Each user is assigned a script according to their computer’s weight. If the prioritized computers are enabled, each Benchmark user is assigned a specific computer to run on. The benchmark status is changed to Start run, with a specification on which run to start.

After a few seconds, the current run information in the cockpit should be updated to reflect the started run. When a run has started, it is possible to stop it by clicking Terminate run. The current step distribution can be activated by clicking on the same button in the cockpit.

Figure 12: Step distribution on the previous page shows an example of the step distribution output. It shows 10 users executing in the Sales order script (SO-BS v2-5). 1 user is currently creating a sales order, 2 users are creating a new line, and so forth. 23 seconds has elapsed since the run started. 22 lines have been created. These figures provide an estimated throughput of 3443 lines per hour. The throughput is calculated more accurately once the run is finished.

Run status

The different run statuses are:

•Withhold

–The run has not yet been started

•Waiting-–The run has been started, but not the user scripts.

•Started-–See step distribution for a progress (activated in Parameters)

•Error-–If a user receives an error, the entire run is terminated (unless Immortal)

•Terminated- the run has been terminated unsuccessfully.

•Finished-–All runs have been successfully completed. This state can only be reached if the Cockpit is open.

The cockpit should be kept open during the run. Every 30 seconds it will recalculate the status of all Benchmark users. If one user encounters an error, the entire run will be terminated unless the immortal flag is set (See Figure 11: Benchmark parameters). When all users are finished executing their user scripts, the cockpit will finish the entire run before calculating the statistics.

The laboratory concept

The Benchmark tool can create a large amount of data within a short period of time, however, the data created might influence the results. Therefore, the data collected by the tool can be exported and imported into a central database for analysis. In order to assure that all data can be imported in the same database and company, the concept of a benchmark laboratory is introduced.

Every Id for a Benchmark run consists of the following:

<Lab Id> <Lab serial number><run serial number>

Laboratory issues are handled in the form Laboratory Registry (See Figure 13: Laboratory registry) This form consists of a Laboratory and a number of series. When a new Benchmark company is created, a laboratory with the name ‘Test’ is created together with its first series.  The first Benchmark run will be called Test0000010001. All test runs should be performed in the test lab. Whenever tests are created that should be saved for later use, the user should create his/her own Laboratory.  At Navision we use the Lab Id to distinguish measurements on IBM, Compaq and HP computers. When the record-breaking test with Compaq was performed in Nice in June 2001, the Lab Id “CPQN” was assigned to all runs. Whenever the database was restored or the course of the tests was changed, the serial number was increased.

@%20=58725AB5=.files/image013.jpg">

Figure 13: Laboratory registry

To partition the runs into groups, the serial number should be used. The statistic results, along with the computer setup associated with a given Laboratory, can be exported and then imported into a central database for analysis. The laboratory registry in Figure 13: Laboratory registry is a snapshot of a central database at Navision.  Results have been imported from various locations so that they can be analyzed across platforms.