By Debra Pacchiano, Vice President of Translational Research and Isabel Farrar, Research Associate at Start Early and Marc Brodersen, Managing Senior Researcher and Joshua Stewart, Senior Researcher at Marzano Research
Whether a tool such as a rubric, assessment, or survey is used to measure teaching practices or academic standards, validation is an important step to ensure users can trust the results. Start Early recently worked with the national education research and consulting firm Marzano Research to examine the validity of a survey developed to support the professional growth and development of early care and education (ECE) staff and administrators.
A growing body of research in ECE and quality improvement shows that strong site-level organizational conditions are key to realizing strong implementation of quality standards and continuous quality improvement in ECE settings. Yet most instruments designed to support the professional growth and development of ECE staff and administrators focus primarily on classroom-level processes, creating a gap that could stifle improvements at the organizational-level related to mindsets and practices surrounding caregiving, teaching and family engagement.
Researchers at Start Early developed The Essential 0-5 Survey to bridge this gap and provide data and actionable information to administrators, teachers, and staff in ECE settings (both school and center-based) serving the families of preschoolers — and now also infants and toddlers. The Essential 0-5 Survey collects teacher, staff and parent experiences and perceptions of the organizationwide mindsets and practices aligned to Start Early’s evidence-based framework of essential conditions: Effective Instructional Leaders, Collaborative Teachers, Involved Families, Supportive Environment, Ambitious Instruction and Parent Voice. Research conducted in both K-12 and ECE settings demonstrates the direct impacts these conditions have on quality practices and children’s immediate and longer-term success. The Essential 0-5 Survey builds on prior research and development efforts between Start Early and the University of Chicago Consortium on School Research, where 5 Essentials survey and evidence-based framework of essential conditions was originally developed and tested in K-12 settings.
To test the relevance of The Essential 0-5 Survey for infant-toddler settings, Start Early conducted 88 cognitive interviews in English and Spanish with teachers, staff and parents from Early Head Start Child Care Partnership programs in two states, which then informed additional revisions. To prepare for pilot study, the refined survey was translated into Spanish, Arabic and Aramaic, and administered to more than 500 teachers and staff and 1,100 parents in California, Colorado, Florida, Illinois, Louisiana, Texas, and Washington D.C.
Validation and Results
Using data from the survey pilot study, Marzano Research conducted a psychometric validation study to understand the following questions:
- Does the survey consistently measure the aspects of organizational quality it was designed to measure?
- Does the survey distinguish sites with different levels of organizational quality?
- Do users with different background characteristics, such as their primary spoken language or the age of the child served by the ECE program, respond to the survey items in similar or systematically different ways?
Pilot study results confirmed the survey items intended to measure particular aspects of ECE organizational quality do group together as intended, and with few exceptions, each grouping measures a singular aspect of ECE organizational quality. Additionally, a majority of the survey items were found to contribute to the overall functionality of the survey; with only a handful of items found to be redundant to each other, or to potentially address different aspects of ECE organizational quality.
These findings speak to the rigor of the survey development team’s process in adapting, writing and vetting the survey items, and importantly endorse the overall psychometric reliability and validity of the surveys for use in infant-toddler center-based settings. However, preliminary analyses also showed that teachers and parents who completed the survey in Spanish or English respond differently to items on several sections of the survey. In some instances, Spanish speakers would generally provide more positive responses to a selection of items while in other instances English speakers would provide more positive responses to a selection of items. Some differences to a selection of items were also found between teachers working with children from birth to age 2 versus those working with children ages 3 to 5.
Next Steps
Start Early is currently using the results of the pilot study to revise items and conduct additional interviews with teachers, staff, and family members of infants and toddlers. Start Early will conduct a formal validation study in 2022 which will include linking survey responses to additional quality outcomes of concern, for example children’s attendance and developmental progress and the quality of teacher-child interactions and family engagement. Learn more about the research behind The Essential 0-5 Survey.
