[ TOC ]

SCORER- AUTOMATIC TEST SCORING SYSTEM REFERENCE CARD

The SCORER system uses the seven-response NAU General Purpose Answer Sheet. These forms must be filled in using a number 2 pencil. Be sure all erasures are complete. The instructor must complete at least one of these sheets (called Point Sheet 1) when using SCORER. The fields on this sheet (Point Sheet 1) must be completed as follows:


COMPLETING TEST KEY (ANSWER SHEET)

LAST NAME, FI, MI- enter the title which you wish to have printed on your SCORER reports.

DEPT NAME- two or three character department code (e.g., CIS).

COURSE NUMBER-three digit course identifier.

VERSION NO- enter the section number for the course. If more than one section is being scored in one batch, this number should be left blank on Point Sheet 1. The system will then pick it up from the individual student forms.

ID NUMBER- enter 9 '1's for Point Sheet 1 (e.g., 111111111).

INST NO- Leave blank.

SEAT NO- Leave blank.

EXAM NO- ALWAYS leave blank, unless you are scoring separate versions of the same exam. If separate versions are being scored, fill in the number of this version (e.g., 1 for the first version, 2 for the second, etc.).

QUESTIONS 1-200- fill in the correct answer to each question. (Note: If your questions do not all have the same point value, you will be using more than one point sheet. Refer to the SCORER User Documentation before filling in any answers).


COMPLETING STUDENT FORMS

Have your students fill out their forms as follows:

LAST NAME, FI, MI- Student's last name, first initial, and middle initial.

DEPT NO, COURSE NUMBER- Leave blank. The system will pick up this information from Point Sheet 1.

SECT NO- if only one section is being scored in this batch, leave this field blank. The system will pick it up from Point Sheet 1. If more than one section is being scored at a time, students should fill in the appropriate section number.

ID NUMBER- student's ID number

INST NO- Leave blank.

SEAT NO- Leave blank, unless the class has specific seating arrangements.

VERSION NO- ALWAYS leave blank unless more than one version of the exam has been distributed to the class. If multiple exam versions are given out, have the students fill in the number of their version.

QUESTIONS 1-200- Students should fill in their choice(s) for each question.


POSSIBLE POINT SHEETS

Point Sheet 1- always required.

Point Sheet 2- (optional) required for tests with at least two different question point values.

Point Sheet 3- (optional) required for tests with at least three different question point values.

Point Sheet 4- (optional) required for tests with at least four different question point values.

Point Sheet 5- (optional) required for tests with at least five different question point values.


CONTROL SHEETS

Control Sheet 6- (optional) used to indicate which subscale each question belongs to.

Control Sheet 7- (optional) used to indicate question types, omitted questions and questions for which all students should be given credit.

Control Sheet 8- (optional) used for penalty scoring.


CONTENTS

Introduction

Section 1 THE FORM (GPAS) ................................... 1 - 1
Section 2 POINT SHEETS ...................................... 2 - 1
Section 3 CONTROL SHEETS ..................................... 3 - 1
Section 4 SUBSCALES .......................................... 4 - 1
Section 5 MULTIPLE ANSWER QUESTIONS ......................... 5 - 1
  SINGLE BUBBLE RESPONSE QUESTION ............. 5 - 1
  ANSWERS OF SAME POINT VALUE ................ 5 - 1
  WEIGHTED ANSWERS .............. 5 - 2
  MULTIPLE BUBBLE RESPONSES- EXACT MATCH ...... 5 - 3
  MULTIPLE BUBBLE RESPONSES- SUBSET ANSWER .... 5 - 4
  ADJUSTED SCORE ................. 5 - 5
  BUBBLE E ........................................... 5 - 6
  OMITTED QUESTIONS .............. 5 - 6
  CREDIT FOR ALL STUDENTS ...... 5 - 6
  SUMMARY OF SHEET 777777777 ................... 5 - 7
Section 6 PENALTY SCORING .................................... 6 - 1
Section 7 PARAMETER SHEET .................................... 7 - 1
Section 8 SCANNING REQUEST FORM .............................. 8 - 1
Section 9 REPORTS ............................................ 9 - 1
  CONTROL SHEET REPORT .......... 9 - 1
  OTHER REPORTS .................. 9 - 2
  GRADE LISTING .................. 9 - 2
  STUDENT REPORT ............... 9 - 3
  *DISTRIBUTION ................ 9 - 4
  *ITEM ANALYSIS ................. 9 - 5
  *SUBSCALES REPORTS ........... 9 - 6
  *EXAM REPORTS .................. 9 - 7
  *OVERALL DISTRIBUTION ........... 9 - 7
  *POSTING REPORT ................. 9 - 7
Section 10 ERROR MESSAGES ...................................... 10 - 1
Section 11 GLOSSARY ............................................ 11 - 1



INTRODUCTION

SCORER is the computerized test-grading system available to the NAU academic community through Information Technology Services.

SCORER was authored in the COBOL language by Mark W. Michrina at the Indiana University in Pennsylvania (IUP), Indiana, PA, in 1982. The basic SCORER system has remained unchanged since then and has been adapted to several different machines.

The Academic Computing (ACAD) group of Information Technology Services maintains SCORER, provides education for academic personnel wishing to use the system, helps to resolve any processing problems and/or conflicts and, when requested and appropriate, captures SCORER data for use by academic personnel. The Operations group provides the primary handling and processing of the tests to be scored.

Despite its simplicity of use SCORER is powerful and diversified. It allows for multiple versions of a particular exam. Common statistics are calculated for each exam version and for the overall exam. They are also calculated for each different class session taking the exam. Questions may be of the single- or multiple-response type. The multiple-response questions can be an exact-match answer or a subset answer. Score adjusting, subscale processing, question omission, and question credit for all students are available. Many question processing options may be weighted. Penalty scoring is available.

Some reports are generated automatically; others must be requested. Reports are available for a class, sections within a class, versions within a class, and version within sections of a class. Report sort options are available.

As you can see, SCORER can take you from the simple to the very complex-your choice.



SECTION 1
THE FORM (GPAS)

NAU's General Purpose Answer Sheet (GPAS) is the form on which students mark their responses to test questions and on which the instructor marks the correct answers - and other data needed in grading a test. This is done by using a pencil with No. 2 or softer lead to darken specific circles on the GPAS. These circles are commonly called "bubbles." GPAS filled out by the instructor with information used for test scoring purposes are referred to as either POINT or CONTROL SHEETS. The GPAS provides the instructor with up to 200 possible questions, each with up to 7 possible answers. The area on the GPAS containing the information that identifies each sheet is divided into fields. There is a field for the NAME, ID NUMBER, DEPT NAME, COURSE NUMBER, INST. NO., SEAT NUMBER, and VERSION- NO. Collectively, these fields are referred to as the Heading Information of the GPAS.

SCORER allows for multiple versions of a particular exam. An instructor may have two or more versions of an exam scored in one batch without separating the answer sheets by indicating in the VERS NO. field in the Heading Information on all POINT and CONTROL SHEETS which exam version each of the SHEETS belongs to (version number and EXAM NO. are used interchangeably in this documentation and have the same meaning). Different exam versions consist of the same exam questions shuffled around into different orders to help reduce the possibility of students cheating. A word of caution - if the instructor scores more than one exam version at a time, all students must mark the correct version number on the GPAS or SCORER cannot function properly. Each exam version must have the same number of omitted questions. The instructor will need to fill out a separate set of POINT and CONTROL SHEETS for each exam version.



SECTION 2
POINT SHEETS

POINT SHEETS are used to indicate to SCORER the point value of a correct answer to a question or, in the case of a weighted question, the point value of a particular response to a question.

A POINT SHEET is indicated by placing one of the following five values in the ID field in the Heading Information.

111111111

222222222

333333333

444444444

555555555

Up to 5 different point values can be used per exam. For example, on a 20-question test where the first 10 questions are worth 2 points each and the last 10 questions are worth 4 points each, the instructor would fill out POINT SHEET 111111111 with the first 10 answers and POINT SHEET 222222222 with the last 10. The instructor then indicates on the Parameter Sheet that POINT SHEET 111111111 has a value of 2, while POINT SHEET 222222222 has a value of 4.

Of all POINT SHEETS, only sheet 111111111 is required. If a test is created in which all questions have the same point value, POINT SHEET 111111111 must be filled out with the correct answers and the Parameter Sheet must indicate the point value.

The instructor may override any student information entered in the following fields by filling the fields with the desired information on POINT SHEET 111111111:

DEPT NAMECOURSE NUMBERSECT. NO.INST. NO.

On POINT SHEET 111111111, SCORER reads all Header Information fields. On POINT SHEETS 222222222 through 555555555, only the ID NO. and the EXAM NO. fields are read; all other Heading Information is ignored.



SECTION 3
CONTROL SHEETS

CONTROL SHEETS are used to tell SCORER how to score the test. A CONTROL SHEET is a GPAS that has a special value in the ID field of the Heading Information. These values range from 666666666 to 888888888. The simplest test has no CONTROL SHEETS, while the most complex has three CONTROL SHEETS. CONTROL SHEETS are used to indicate in which subscale(s) a particular question is to be scored, which type of multiple-answer question is being scored, whether a question is to be omitted from the test scoring process, whether all students are to be given credit for a question, or how many choices there are for each question that is involved in penalty scoring.

SCORER reads only the ID field and the EXAM NO. field of the Heading Information on all CONTROL SHEETS. Instructors may fill in the other fields for their own use.

Sections 4, 5, and 6 will discuss the three types of CONTROL SHEETS and their uses.



SECTION 4
SUBSCALES

Subscales can be used to tally student responses to different types of questions. For example, an instructor can determine from using subscales if students perform better on factual or conceptual questions, true/false or multiple choice questions, questions based on text or on the lecture, questions based on older material or newer material, etc. Subscales can also be used to measure the validity of a test by comparing the results of the first half of the test with the last half, the even-numbered questions with the odd-numbered ones, etc.

Subscales are indicated by using a CONTROL SHEET which has 666666666 in the ID field of the Heading Information. There are eight different subscales available for each question which correspond to the letters A through G and the blank (no bubbles darkened). For a question to be tallied in a particular subscale, the subscale for that question must be marked on CONTROL SHEET 666666666 at that question number. For example, if question number one is to belong to the two different subscales A and F, then for question one on CONTROL SHEET 666666666 bubbles A and F should be marked.

Reports are available for subscale breakdowns by section, exam, version, and by the individual student's performance. To get SCORER to print the reports the instructor must indicate on the Parameter Sheet that they are desired.



SECTION 5
MULTIPLE ANSWER QUESTIONS

SCORER provides for scoring multiple-answer questions. This offers a variety of options in creating a test to suit all of an instructor's needs. To score multiple-answer questions, a CONTROL SHEET is filled out containing 777777777 in the ID field of the Heading Information. A question marked on this sheet indicates one of the following three things:

  1. This is a multiple-answer question:
    1. with a Single Bubble Response     (A)
    2. with Multiple Bubble Response-Exact Match     (B)
    3. with Multiple Bubble Response-Subset     (C)
    4. with Multiple Bubble Response-Adjusted Score     (D)
  2. This question is to be omitted     (F)
  3. All students are given credit for this question     (G)

If a 777777777 sheet is included with the test to be scored and a question is not marked on the 777777777 sheet then for that question, only one answer may be bubbled on the POINT SHEET, i.e., if a 777777777 sheet is included, not all questions need to be of the MULTIPLE ANSWER type.



SINGLE BUBBLE RESPONSE QUESTIONS

ANSWERS OF THE SAME POINT VALUE

The first type of multiple-answer question limits the student to a single bubble response. If there is more than one correct answer to a question with each answer consisting of only one bubble and each answer being worth the same points, the instructor would mark all of the correct answers on a single POINT SHEET and darken in the A bubble on the CONTROL SHEET 777777777. In this manner, a student marking any ONE of the correct answers is awarded the specified amount of points.



WEIGHTED ANSWERS

If each answer to a multiple-answer question with a single bubble response is to be awarded a different point value, the question is said to be weighted. To fill out the CONTROL SHEETS for this type of question, the instructor must darken the bubbles on the POINT SHEETS (111111111 through 555555555) which correspond to the weighted responses and CONTROL SHEET 777777777 must have bubble A darkened for that question. Bubble A indicates that this question is of a single bubble response type. For example, if question five is a weighted question with response A worth five points, response C worth 3 points, and response F worth 1 point, the POINT SHEETS and the CONTROL SHEET will be filled out as follows: on POINT SHEET 111111111, bubble F would be darkened for question five; on POINT SHEET 222222222, bubble C would be darkened for question five; on POINT SHEET 333333333, bubble A would be darkened for question five, and on CONTROL SHEET 777777777, bubble A would be darkened for question five. (The point value to be assigned to POINT SHEETS 111111111 through 333333333 is handled on the Parameter Sheet). This combination of four sheets indicates that for question five: if the student marks bubble A, 5 points are awarded; if the student marks bubble C, 3 points are awarded; if the student marks bubble F, 1 point is awarded. Any other response by the student, or multiple responses, yields zero (0) points. See the following example.

      5. The National Debt in 1981 was:
  A. Close to $1 Trillion B.Zero
 C. 30% of the GNP D.Five dollars
 E. Six Dollars F.Very Large

Here the National Debt was indeed approximately $1 trillion. Therefore, answer A is awarded 5 points. 1 Trillion dollars was also 30% of the GNP, so answer C is awarded 3 points. The National Debt was also very large, so answer F is awarded 1 point. Keep in mind that a student may mark one and only one response on this type of question.



MULTIPLE BUBBLE RESPONSES-EXACT MATCH

The second type of multiple-answer questions is a question for which each possible answer requires a combination of two or more bubbles. These answers can also be weighted by placing the different answers on different POINT SHEETS. To indicate this type of question, the B bubble is marked on CONTROL SHEET 777777777 for this question. For example, if B & C is worth 3 points, and C & D is worth 1 point, the POINT SHEETS and the CONTROL SHEET would be filled out as follows: bubbles B & C would be darkened on POINT SHEET 111111111; bubbles C & D would be darkened on POINT SHEET 222222222; and, bubble B would be darkened on CONTROL SHEET 7777777777. See the following examples.

    1. Which two of the following items make up the best breakfast combination?
  A. Macaroni B.Bacon
 C. Eggs D.Toast

Here, B & C are the desired answers, so they are worth 3 points. However, if the student marked C & D, 1 point would be awarded because this breakfast is acceptable, but is not as good as B & C.

    2. Match the author in Column 1 with the novel in Column 2.
      (1)     (2)
  01) Dickens AB) Three Musketeers
  02) Twain AC) Grapes of Wrath
  03) Hawthorne AD) Madam Bovary
  04) Fielding AE) Huckleberry Finn
  05) Dumas AF) Tess of the D'ubervilles
  06) Steinbeck AG) Golden Bowl
  07) Hardy BC) Farewell to Arms
  08) Hemingway BD) Oliver Twist
  09) Balzac BE) Tom Jones
  10) James BF) Scarlet Letter

This is an example of a set of matching questions that show when two letters would be necessary for a correct answer. This type of question also shows that weighting is not necessary for a question of this type. Using two letter responses, a maximum of 21 choices in a single matching list may be created. Using three letter responses, a maximum of 35 choices in a single matching list may be created.



MULTIPLE BUBBLE RESPONSES-SUBSET ANSWER

The third type of question is one where the student must answer the question by darkening in a subset of the bubbles that the instructor has darkened. Any bubble that the student darkens that the instructor hasn't results in zero (0) points for the student for that item. The instructor may mark a group of bubbles on each POINT SHEET. To indicate a question of this type, bubble C is darkened on CONTROL SHEET 777777777. For example, students will receive 5 points if they darken in bubble A, B, or C, or any combination of A, B, & C, or the students will receive 1 point if they darken in bubble D, E, F or G, or any combination of D, E, F & G, the POINT SHEETS and the CONTROL SHEET 777777777 would be filled out as follows: bubbles A, B, & C must be darkened on POINT SHEET 111111111 and D, E, F & G must be darkened on POINT SHEET 222222222 and C must be darkened on CONTROL SHEET 777777777. See the following example.

    1. Choose the cities that have a population under 100,000, or choose the cities with a population of over 1,000,000.
    A.   New York City, NY B. London, England
    C.   Chicago, IL D. Clymer, PA
    E.   Portage, PA F. Homer City, PA
    G.   Cassandra, PA

Here, a correct answer is considered to be any combination of A, B & C, or any combination of D, E, F, or G. If a student answers B, it is a correct answer worth 5 points. If another student answers D, E, & F, it is a correct answer worth 1 point. However, if a student answers, B, C, D, & E, it is an incorrect answer and is awarded no points.

This type of question may be weighted with different subsets being worth different point values. Subsets are not permitted to overlap across POINT SHEETS.



ADJUSTED SCORE

The last type of multiple-answer question is the Adjusted Score. For this question, the student is awarded X number of points for a bubble darkened that matches a bubble darkened by the instructor and X number of points is subtracted for each bubble the student darkens that the instructor has not darkened. The lowest number of points the student can get for this type of question is zero (0). X number of points is determined by the POINT SHEET on which the instructor has marked his bubbles. This type of question is limited to one POINT SHEET. This type of question is indicated by darkening the D bubble on CONTROL SHEET 777777777 for the particular question. See the following example.

    1. Identify any courses satisfying Liberal Studies requirements for a degree from a typical university.
    A.   Marching Band B. Intro to Economics
    C.   General Psychology D. Advanced Electronics
    E.   Intro to Theater F. Child Development
    G.   Family Planning

There are three correct answers, B, C, & E. If the instructor darkened these bubbles on POINT SHEET 111111111 and assigns a point value of 1 for this POINT SHEET, the maximum number of points a student can earn for this question is 3. The least points is zero (0). For instance, if a student darkens bubbles A, B, & C, the student is awarded 1 point because the answer that the student darkened incorrectly cancels one of the correct answers darkened.



BUBBLE E

This is not used.



OMITTED QUESTIONS

For a question to be omitted, the instructor must darken bubble F on CONTROL SHEET 777777777 for that question. This question is then treated as if it never existed.



CREDIT FOR ALL STUDENTS

For all students to be given credit for a question, the instructor must darken bubble G on CONTROL SHEET 777777777 for that question. If no bubbles for a question are marked on any of the POINT SHEETS, no points will be awarded. If bubbles are marked on two or more POINT SHEETS the highest point value is awarded.



SUMMARY OF SHEET 7777777

Summary of possible responses on CONTROL SHEET 7777777:

  1. Single bubble responses with multiple correct answers. Weighted answers using different POINT SHEETS or non-weighted answers using only one POINT SHEET.
  2. Multiple bubble responses (exact match) - may be weighted using more than one POINT SHEET.
  3. Multiple bubble responses (subset) - may be weighted.
  4. Adjusted score answers.
  5. NOT CURRENTLY USED.
  6. Omitted question.
  7. Credit for all students.



SECTION 6
PENALTY SCORING

Penalty scoring is allowed on questions that are not of the multiple answer type, i.e., CONTROL SHEET 777777777 may not be marked for a question on which penalty scoring is to be done. Penalty scoring is a method of scoring that attempts to eliminate guessing on the student's part by penalizing all wrong answers but not penalizing any answers that have been left blank (omitted). The penalty for a wrong answer is calculated as follows:

1 / (number of choices for this question - 1) * POINT SHEET value for the question

The number of choices for each question is necessary in calculating the penalty. The number of choices for each question is indicated by another CONTROL SHEET that has 888888888 in the ID field of the Heading Information. For each question that has 4 choices, bubble D should be marked for the question if penalty scoring is desired (because D is the 4th bubble). If the question has 6 choices, mark bubble F (the 6th bubble). SCORER uses CONTROL SHEET 888888888 to determine the penalty for a particular question. When CONTROL SHEET 888888888 is not present, or if CONTROL SHEET 888888888 is present but no bubbles have been marked for a particular question, SCORER assumes that no penalty is to be assessed for an incorrect answer to that question.




SECTION 7
PARAMETER SHEET

The Parameter Sheet contains information that SCORER needs to function. This information must be filled in by the instructor.

The parameter information is entered into the system by the staff of Information Technology Services from the information indicated on the Parameter Sheet. A copy of the Parameter Sheet is given with each set of documentation for SCORER. Please make copies of this form. Any exam brought for SCORER processing that is not accompanied by a properly completed Parameter Sheet (and Scanning Request Sheet) cannot be processed and will be returned to the instructor.

The Parameter Sheet information items and their possible values are:

Column No. Description Values
1-3 Number of the first question to be scored. 1 to 200
4-6 Number of the last question to be scored. 1 to 200
7-9 Total number of questions to be omitted. 1 to 200
10-11 Number of exam versions to be scored. 1 to 200
12 Sort order for instructor's grade listing. (1) for Name
    (2) for SocSecNum
    (3) for Seat
13-14 Point value for POINT SHEET 111111111. 1 to 99
15-16 Point value for POINT SHEET 222222222. 1 to 99
17-18 Point value for POINT SHEET 333333333. 1 to 99
19-20 Point value for POINT SHEET 444444444. 1 to 99
21-22 Point value for POINT SHEET 555555555. 1 to 99
23 ??? Missing from printed doc. Y or N (?)
24 Option to print grade distribution statistics for each class section. Y or N
25 Option to print item analysis for each class section. Y or N
26 Option to print subscale scores for each class section. Y or N
27 Option to print grade distribution statistics for each exam version. Y or N
28 Option to print item analysis for each exam version.

Y or N
29 Option to print subscale scores for each exam version. Y or N
30 Option to print overall grade distribution statistics for all exam versions. Y or N
31 Option to print posting list by sections. Y or N



SECTION 8
SCANNING REQUEST FORM

The Scanning Request Form must be filled out by the instructor and placed on top of the packet of forms to be processed by SCORER. Always check the SCORER box at the top-center of the form and fill in the spaces for NAME, PHONE (where you can be contacted should a problem/processing-delay occur), DATE NEEDED (see discussion below), COURSE and SECTION(S).

Please count ALL the GPAS-this includes the student forms, the POINT SHEETS and the CONTROL SHEETS and the TOTAL # OF BLUE FORMS!!! This is very important. DO NOT guess. COUNT!!! Enter this count on the Scanning Request Form where it says "FORM COUNT." This count will be verified by Information Technology Services when the forms are scanned.

The DATE NEEDED area of the Scanning Request Form is an optional field as the turn-a-round time for a regular SCORER run is 24 hours. This means that if it is at all possible, you can pick up the forms and reports generated by SCORER 24 hours after they are delivered to Information Technology Services. If there are power or equipment failures or problems with the Parameter Form data, POINT SHEETS or CONTROL SHEETS, there could be a delay in meeting our 24-hour service goal.

During finals week when your deadline for submitting grades is most critical, we direct our energies toward a 4-hour turn-a-round. Again, there are things that we cannot anticipate or prepare for that can modify our service goals for you.

You also receive a copy of the Scanning Request Form with each set of documentation. Please make copies of this form, also.




SECTION 9
REPORTS

CONTROL SHEET REPORT

For the instructor's convenience, and also for clarity, SCORER produces a report that indicates exactly how it interpreted the CONTROL and POINT SHEETS filled out by the instructor. If there are any problems with the CONTROL and/or POINT SHEETS, the errors are indicated for each question. Although the instructor will probably never see any of the error messages, as they are corrected before the SCORER reports are released to the instructor, a legend of the messages is given at the end of this documentation.

When the instructor receives the reports the only messages that should be given on the CONTROL SHEET REPORT are the ones denoting special process handling as signified on the CONTROL and POINT SHEETS, example: *12 All students have been given credit for this question.



OTHER REPORTS

SCORER produces several other reports in addition to the CONTROL SHEET REPORT. These other reports are described in the following paragraphs in the order in which they are printed. Any optional report is marked with a star (*). Optional reports are not generated or printed unless the instructor indicates on the Parameter Sheet that they are wanted.



GRADE LISTING

This listing is for the instructor's reference and contains information on each student's exam performance. The students are listed in one of three orders as selected by the instructor on the Parameter Sheet. Other information for each student follows:

Number Right - This is a tally of the number of questions for which each student received points. In the case of a weighted question, a response that earned any points for the student is tallied as a right response.

Number Wrong - This is a tally of the number of questions for which each student received no points. Number Right + Number Wrong + Number Omitted = Total test questions.

Number Omitted- This is a tally of how many questions to which each student did not respond. This does not include questions indicated by the instructor to be omitted.

Points Score - This is the student's raw score in points

Percent Score - This is the student's percent score which is calculated by dividing the Point Score by the total possible points and multiplying by 100. This calculation is then rounded.

If a student marks an invalid EXAM NO. on the GPAS, a message will be printed next to that student's name instead of the score. The message reads "Bad EXAM NO. This test not scored."



STUDENT REPORT

This report includes details on each student's individual performance. Each student is listed on a separate page in order to allow for easy distribution to the students, if desired. The report includes the student's name, identification number, number of questions answered correctly, number of incorrect answers, total points scored on the test, and percent score. It lists the questions that the student missed with both the student's answer and correct response. If subscales are used, the student's score on each subscale will be included on the report.



* DISTRIBUTION

A section distribution report contains a histogram of the text scores to visually represent how the class section performed on the test. Statistics on the left-hand side of the report are:

Number of students: A tally of the number of students in this section who took the exam. This does not include students who indicated an invalid EXAM NO.
Sum of Scores: This is a tally of all the student scores.
Sum of Scores Squared: This is a tally of the student scores squared.
Mean: This is a calculation of the mean percent score of all students in the section. This value is rounded.
Standard Deviation: This is a calculation of the deviation from the percent score. This calculation is also rounded.

The histogram will scale itself if more than fifty students score a particular percentage. This scale is at the bottom left-hand side of the report. Each star (*) on the histogram indicates one occurrence of that percent score unless otherwise indicated by the scaling number. The number above each column of stars is a count of how many stars are in that column.



* ITEM ANALYSIS

This report gives information on each question, particularly information on each possible response. For each bubble, the following data is supplied:

X:Y (W)
          ------     where
Z

X= tally of students in this section darkening this bubble on their GPAS.

Y= the percentage of total students in this section who darkened this bubble.

Z= the average overall score of all students marking this bubble.

W= which POINT SHEET (if any) this bubble was marked on.

A correlation is calculated for each question. The correlation is between the students' performance on each particular question with how well the students performed overall. The formula used to calculate this correlation is the Pearson Product-Moment Correlation Coefficient. It was obtained from "Statistical Analysis in Psychology & Education" by George A. Ferguson.

R# =

N XY - XY
--------------------
v[NX-(X)] [NV-(Y)]

If the correlation is near the value of the positive one (1), then students performed as well on this question as they did overall (closer the value is to one (1), the better a question represents the exam). If the correlation is near the value of minus one (-1), then the students' performance on this question is inversely representative of their overall score (closer the value is to minus one (-1), the worse this question represents the exam).



*SUBSCALES REPORT

The section subscale report provides information on a student's performance on each subscale that was indicated. For each subscale that was indicated, the following information is supplied:

Possible Points: This is a computer-calculated, highest-score-possible number given for reference.
Mean (Points): This is the raw score mean calculated for each subscale.
Mean (Percent): This is the percent score mean calculated for each subscale.
Standard Deviation (Percent): This is the deviation from the percent score calculated for each subscale.
Range: This is the range of raw scores for each subscale. The range is calculated by subtracting the lowest raw score from the highest raw score for each subscale. The total column of the range represents the range for the entire test.



* EXAM REPORTS

The next three reports are exam distribution, exam item analysis, and exam subscales, respectively. These reports are identical to the section reports of the same type, except that the word "ALL" is printed in the section field on the top of each report. The EXAM REPORTS should be requested instead of the section reports if only one class section is being scored.



* OVERALL DISTRIBUTION

The overall distribution report is identical to the other distribution reports with the exception that it represents a distribution of all exams scored in the batch, regardless of SECTION NUMBER and EXAM NO. An exam number of "ALL" identifies this report. This report should be requested only if there is more than one exam version that is being scored.



* POSTING REPORT

This report is a listing of the student scores. It includes each student's point score in addition to the percent score and the total points possible for the exam. Also, all student grades for all sections and exam versions are listed on one report which cuts down on the number of listings an instructor must post.




SECTION 10
ERROR MESSAGES

All error messages generated by SCORER are listed below. The number before each message is the error message number. Error messages preceded by one star (*) do not terminate a SCORER run. Error messages preceded by two stars (**) cause SCORER to terminate without scoring the exam.

**1 For a question only allowed to have a single answer (a single bubble), more than one bubble is marked on POINT SHEET.

**2 More than one bubble has been marked on CONTROL SHEET 888888888 for this question.

*3 More than one subscale group was marked for this question. This is not an error but a warning to the instructor that more than one subscale has been marked. Program function will continue.

**4 A specific bubble was marked on a combination of POINT SHEETS. That is, marked bubbles are overlapping.

**5 More than one bubble is darkened on CONTROL SHEET 777777777 for this question.

**6 CONTROL SHEET 888888888 is marked for a question on which penalty scoring is not valid. Penalty scoring is only valid for questions that have not been marked on CONTROL SHEET 777777777.

**7 For a single answer question, more than one bubble has been marked on a combination of POINT SHEETS.

**8 For question type B, the exact same bubbles have been marked on more than one POINT SHEET.

**9 For question type D, more than one POINT SHEET was marked.

*10 An invalid bubble has been marked on CONTROL SHEET 777777777. Bubble E is not currently used. Program continues to run.

*12 All students have been given credit for this question. In the case of weighted questions, the highest weight value is awarded. Only a reminder.

**13 The number of omitted questions on CONTROL SHEET 777777777 does not match the number of omitted questions on the Parameter Sheet.

*14 No bubbles have been marked on any of the POINT SHEETS for this question. The question is considered to be omitted.

**15 POINT SHEET 111111111 has not been found for the exam that is being processed.

**16 Running of SCORER has been halted by one or more fatal errors.

**17 Number of test versions found by SCORER does not match the number indicated on the Parameter Sheet.

*18 A request has been made on the Parameter Sheet for subscales to be printed, but CONTROL SHEET 666666666 has not been found. Request is ignored but SCORER continues to run.

**19 The input file from the scanner (SCANNER-FILE) is empty.

**21 First question value (1-3) on parameter card is not numeric.

**22 First question value (1-3) on parameter card is less than 1.

**23 First question value (1-3) on parameter card is greater than 200.

**24 Last question value (4-6) on parameter card not numeric.

**25 Last question value (4-6) on parameter card is less than 1.

**26 Last question value (4-6) on parameter card is greater than 200. This is not possible.

**27 First question value (1-3) is greater than the last question value. This is not possible.

**28 Number of omitted questions (7-9) is not numeric.

**29 Number of omitted questions (7-9) is greater than 200.

**30 Number of omitted questions (7-9) is greater than the number of questions on the test. Number of questions on the test is calculated by subtracting the number of the first question from the number of the last question and adding one (1).

**31 Number of test versions (10-11) is not numeric.

**32 Number of test versions (10-11) is less than 1.

**33 Sort order value (12) is illegal. Legal values are 1, 2, or 3.

**34 Point value for sheet 111111111 (13-14) is not numeric.

**35 Point value for sheet 222222222 (15-16) is not numeric.

**36 Point value for sheet 333333333 (17-18) is not numeric.

**37 Point value for sheet 444444444 (19-20) is not numeric.

**38 Point value for sheet 555555555 (21-22) is not numeric.

**39 Individual subscales indicator (23) not "Y" or "N".

**40 Section distribution indicator (24) not "Y" or "N".

**41 Section item analysis indicator (25) not "Y" or "N".

**42 Section subscales indicator (26) not "Y" or "N".

**43 Exam distribution indicator (27) not "Y" or "N".

**44 Exam item analysis indicator (28) not "Y" or "N".

**45 Exam subscales indicator (29) not "Y" or "N".

**47 Parameter file not found.

**52 Duplicate CONTROL or POINT SHEET found for a particular exam version.

**54 Overall distribution indicator (30) not "Y" or "N".

**55 Section posting indicator (31) not "Y" or "N".




SECTION 11
GLOSSARY

This glossary has been supplied to aid in defining terms that users of SCORER may not be familiar with. The listing is in alphabetical order.

111111111 - A POINT SHEET. The point value for questions marked on this sheet must be indicated on the SCORER Parameter Sheet.

222222222 - A POINT SHEET. The point value for questions marked on this sheet must be indicated on the SCORER Parameter Sheet.

333333333 - A POINT SHEET. The point value for questions marked on this sheet must be indicated on the SCORER Parameter Sheet.

444444444 - A POINT SHEET. The point value for questions marked on this sheet must be indicated on the SCORER Parameter Sheet.

555555555 - A POINT SHEET. The point value for questions marked on this sheet must be indicated on the SCORER Parameter Sheet.

666666666 - A CONTROL SHEET. Any question marked on this sheet has been marked to be scored in a subscale. There are 8 possible subscales, A through G and blank.

777777777 - A CONTROL SHEET. Any question marked on this sheet is marked for one of the following possibilities:

  1. Multiple response question.
  2. Omitted question.
  3. All students given credit.

888888888 - A CONTROL SHEET. Any question marked on this sheet is to be scored with the penalty scoring option. The bubble marked corresponds to the number of choices the question has.

ANSWER - An answer is a marked bubble, or a series of marked bubbles, that the instructor indicates which are used in correcting tests to determine values.

BUBBLE - A small circle on the GPAS is considered a "bubble".

CONTROL SHEET - This is a GPAS that has a special value in the ID field of the Heading Information. Possible values and their significance are:

666666666 - indicates specific subscale(s).

777777777 - indicates multiple-answer questions.

888888888 - indicates # of choices per question.

EXAM NUMBER - Field of the GPAS Heading Information. It is used as an exam version field. If only one exam version is scored at a time, this field may be left blank. The term EXAM NO. is synonymous with VERSION NUMBER.

GENERAL PURPOSE ANSWER SHEET - This is an NAU form used with SCORER. The form is blue ink printed on white paper and consists of two parts, the Heading Information (including name, sect. etc.) and the answer area (which provides possibilities for up to 200 questions, seven possible responses per question).

GPAS - An acronym for "General Purpose Answer Sheet."

HEADING INFORMATION - This is an area of the GPAS that identifies the SHEETS from each other. There is a field for name, dept., course no., section, id number, inst. number, seat number, and exam version.

PARAMETER SHEET - This is a form which must be filled out by the instructor and is used in running SCORER.

PENALTY SCORING - This is a special method of scoring exams that attempts to eliminate guessing on the part of students.

POINT SHEET - This sheet determines how many points are to be awarded per question. Possible point sheets are:
    111111111
    222222222
    333333333
    444444444
    555555555

An answer marked on one of these sheets indicates how many points a student is to be awarded if the response matches the answer. Those points are awarded according to the value of that particular POINT SHEET, as indicated by the instructor on the Parameter Sheet.

RESPONSE - This is what the student makes as an attempt to answer a question correctly.

SUBSCALES - Subscales are parts of a test on which an instructor wants separate statistics.

VERSION NUMBER - Same as the EXAM NO.


SCORER User Documentation Revision 3.1     April, 2015