Resources Available for Student Ratings
Explorance Blue Website Learn more about Explorance Blue
Fresno State Student Rating of Instruction (FSSRI)
Answers to Faculty Questions
Why did we stop using IDEA?
When our contract with IDEA ended, the Academic Senate decided to look into other options rather than automatically renewing it. IDEA was about to make major changes to its administration, including eliminating the option for paper questionnaires. Proposals were solicited from companies that provide this service, and a task force was created of Fresno State experts to propose an instrument created specifically for our needs. In the end, the Senate voted that the Fresno State instrument was a better match to our needs than others in the running, including the IDEA instrument.
What is the new instrument?
This new instrument, the Fresno State Student Ratings of Instruction (FSSRI), was designed by a task force comprised of faculty representatives from all 8 colleges, some with professional expertise related to the assessment of learning as well as scale construction. It is based on the principle that we should ask students to report on directly observable instructor behaviors for which there is published empirical evidence that the behavior promotes learning.
The specific items are available for review on the faculty affairs website, http://fresnostate.edu/academics/facultyaffairs/documents/teaching-effectiveness/FSSRI_Question_Items.pdf, subject to approval by the Academic Senate and the Provost, which we anticipate in September 2019.
Does it meet the requirements of our APM?
Absolutely. The APM requires several things:
1. Assess these three dimensions of teaching: instructional design, instructional
delivery, and assessment. The FSSRI includes four items about each dimension for a
total of 12 items. We selected the topics of the four items by reviewing the published
empirical literature on effective teaching and learning and identifying those practices
that are known to promote learning.
2. Allow departments to customize questionnaires by choosing from a pool of items approved by the Academic Senate and the Provost. The FSSRI has essentially 12 small pools of questions. This assures that all of the mandated dimensions (see point #1) are included. Each pool of items has a default item, but faculty who use an online questionnaire may customize by replacing that item with another from the same pool of items, or by adding even more items from that pool.
The instrument must be reliable and valid. We conducted extensive pilot testing in Spring 2019 on the new instrument. That project firmly established the reliability and validity of this tool. While additional research can (and should) be done to investigate additional forms of reliability and validity and to make improvements, we have done more than most proprietary student ratings instruments. Full details are available upon request.
How will I customize my student ratings questionnaire?
On November 4, instructors of every course in the system will receive an email notification that they can log in to the Explorance Blue system. You can follow the link in the email, or you can go into the Student Ratings link that will appear in your Canvas courses. Once you get to your Explorance portal, (Explorance is the company that will be administering the questionnaires for us) you have three choices:
- Opt OUT of evaluation for that course if it is not necessary by your department policy. This option will disappear after Nov 8.
- Select the PAPER option, in which case you will not be able to customize the questionnaire. This will automatically generate a PDF of your survey, which you need to download and email to your DAA. This option will disappear after Nov 8.
- Select the online option, and designate the dates that the online questionnaire will be available to your students. Starting on November 9, you will also be able to customize your questionnaire using the available items. You must have at least one item in each of the 12 categories, so at least 12 items. You may add items, up to a limit of 20 total. You may add written questions in addition to the standard items.
Can I still administer my student ratings on paper, or do I have to go online?
Paper surveys are currently still an option, but the extra cost of paper surveys is prohibitive, and we may eliminate this option a few years down the line. For now, when an individual faculty member chooses to administer surveys on paper, that person’s department office will be responsible (both financially and in terms of labor) for printing the surveys. Also, it will not be possible to customize paper surveys by swapping out or adding items.
What if I teach a lab class? Or something else that doesn’t fit the typical classroom setting?
The trick to high response rates with student ratings surveys is to administer them during the protected time in class, just as faculty do with paper surveys. Students will be able to access the surveys on their phone/laptop/tablet in class when you tell them it’s time to do so. Anyone not in class that day can go online later to complete the survey if they wish. When this method is used, the drop in response rates is minimal.
Another strategy that we will employ is to embed our student ratings system into Canvas. Therefore, whenever a student goes to Canvas, he or she will see an automatic pop-up reminder to complete the surveys that have not yet been completed. Furthermore, faculty will be able to monitor, in real-time, what their response rate is. (Not WHO has responded, or see the completed surveys of course, but just how many students have responded. These strategies will be very useful to those who are not able to administer the surveys in class.
Are there special items for online classes?
No. Online courses will be evaluated by the same criteria as face-to-face classes. We did verify that the wording of most items will work on the online context
How will students access the online student ratings questionnaires?
Students will be invited by email to complete their questionnaires. They will also have reminders pop up in Canvas. Both will provide a link to the Explorance webpage where they can find the questionnaires for their classes.
How will I get a good response rate with online student ratings questionnaires?
The approach that yields high response rates is to administer the student ratings during the protected time in class, just as faculty do with paper questionnaires. Students will be able to access the questionnaires on their phone/laptop/tablet in class when you tell them it’s time to do so. Anyone not in class that day can go online later to complete the questionnaire if they wish. When this method is used, the drop in response rates is minimal.
Another strategy that we will employ is to embed our student ratings system into Canvas. Therefore, whenever a student goes to Canvas, he or she will see an automatic pop-up reminder to complete the questionnaires that have not yet been completed. Furthermore, faculty will be able to monitor, in real-time, what their response rate is. (Not WHO has responded, or see the completed questionnaires of course, but just how many students have responded.) These strategies will be very useful to those who are not able to administer the questionnaires in class.
Can I identify the date when student ratings questionnaires are available to my students?
Yes. When you go into your Explorance portal on Nov 4 and select online administration, you will be able to designate dates, so long as they are between Nov 25 and Dec 11. Ideally, you would open them the day that you intend to provide time in class and keep them open for a student who missed class that day.
Do online student ratings administered in class still need a proctor?
According to the APM, yes. If the Senate wants to reconsider the policy (which is already nonsensical for online classes), they can consider this issue at that time. But with online questionnaires completed in class, the job of the student “proctor” will probably be just to alert the instructor when everyone is finished.
What will happen if I do nothing?
If you do nothing, then your course will have an online instrument created that will consist of the 12 default items. Your students will be notified by email and in Canvas on Nov 25 that the survey is waiting to be completed, and it will close on December 11. Afterward, the report will be generated and will be available to you and your Dean’s Office to be placed in your personnel file.
What if I don’t need to administer student rating surveys in my course this semester?
The new system is going to automatically generate questionnaires for every course in the system. So you will have to opt OUT if you do not want to evaluate a class. Please note that this is the opposite of what used to happen. In the past, packets were only prepared if you opted IN. If you are going to opt OUT, look for the email that should arrive on Nov 4 alerting you that it’s time to select how, or if, you will administer the ratings questionnaire in your class. Follow the link, or go into Canvas to find the link, and turn OFF evaluations for the specific course that does not need to be evaluated. Note that department chairs will be notified about who has opted out to verify that it is allowed by the university and department policy.
How will I get my reports after the semester is over?
Score reports will be available in the faculty Explorance portal after the grades due deadline. Chairs and deans offices will have access to the static reports for review and for placement in personnel files. You will get an email inviting you to access the reports in your Explorance portal when they are ready. They will be available for classes evaluated online before they will be available for classes evaluated on paper.
How are my scores likely to compare to my IDEA ratings?
Like IDEA ratings, the FSSRI ratings are skewed positively (mean score is 4.1 on a 5-point scale). The correlation with IDEA is .80. So your scores are likely to be very similar. Out of the 67 classes in the pilot study with both IDEA and FSSRI data, 24 had identical scores on the two instruments. Another 21 had an FSSRI score that was lower by 0.2 of a point, and 11 had an FSSRI score that was higher by 0.2 of a point. Only 2 classes had a score that differed by 1 point (on the 5-point scale) and those both scored higher on FSSRI than on IDEA. So we don’t expect any surprises with the data produced by the new instrument. We just expect that they will be more useful because instructors will see exactly what they can do to improve because it’s obvious from the items.
What are the dates that things will happen in Fall 2019?
- Oct 1: Department Administrative Assistants who may need to print paper questionnaires should register for an account at the Print Shop. All paper questionnaires should be printed centrally so that they are standard and can be “read” by the scanning software after they are completed. Register here: https://printshop.fresnostate.edu
- November 4: Faculty get access to their Explorance portal. For each class, faculty will choose to opt-out, or to administer on paper, or to administer online.
- November 8: Decisions to opt-out or to use a paper questionnaire must be in the system by this day. Those options will no longer be available.
- Nov 9 thru Nov 24: Faculty can customize the items on their student ratings forms.
- November 25: Student rating questionnaires go live unless the instructor has chosen to delay it.
- Dec 11: Last day of instruction, student rating questionnaires close unless the instructor has selected an earlier date.
What is known about the validity of online student ratings?
- University of Oregon's information about online student ratings and other information: https://registrar.uoregon.edu/course-evaluations/response-rates-and-accuracy
Who can answer questions about this?
Talk to your college representative on the Task Force, or direct questions to the chair of the committee, Katie Dyer.
- Arts & Humanities: Natalie Munoz
- Craig School: McDowell Porter
- Health & Human Services: Gyanesh Lama
- Jordan College: Katie Dyer & Athanasios Alexandrous
- Kremen School: Monica Billen
- Lyles College: Nagy Bengiamin
- Social Sciences: Tom Holyoke
- Science & Math: Dermot Donnelly
What are best practices for the interpretation and use of student ratings data?
Link to a literature review and analysis focused on how administrators and personnel committees should use student rating data.