The Most Fascinating Article You’ll Ever Read About Educational Assessment

In the 1970s, Caroline Fahmy’s father started a company. Some things have changed.

INTERVIEW | by Victor Rivero

CIRCA 1985. What were you doing in the mid-1980s? Mark Zuckerberg was barely a year old, Ronald Reagan was mid-presidency, the CD-ROM was an emerging technology, and student-to-computer ratios in schools averaged 1-to-92. Without email, computers were a step above typewriters and looked like the one sitting on Caroline Fahmy’s desk in this photo, taken at the office of Educational Data Systems, the company she worked at then—and now leads today. She’s seen a lot—and shares more in this interview.

Not many people can say they truly grew up in a data-driven household.

In 1974, Caroline Fahmy’s father started Educational Data Systems, a company that prides itself on providing accurate assessment data to districts across the country.

In 1996, she took over as president and continues to uphold her father’s legacy.

For the last eight years, the Morgan Hill, Calif.-based company with just over 30 employees (more than half have been with the company 10+ years) has been at the core of processing data for California English Language Development Test (CELDT), a paper-based assessment that 1.5 million ELLs have taken every year.

Testing, assessment, technology, learning, student data, and many other issues are all discussed here in quite some depth—a depth that Caroline is at ease with, as she’s lived it for the greater part of her career.

So, let’s just start at the beginning and see what we can learn.

And, promise: it won’t take nearly as long as what a student might endure at their desk with a No. 2 pencil.

When did you start working for your father at Educational Data Systems?

I graduated from college with a degree in art history and, after many months of applying and being turned down for various jobs, I asked my dad if he needed help running his business. That was December 1981, and I haven’t looked back. When our bookkeeper left, I took over the financial responsibilities for the company. After five years of additional college courses, I received my MBA with a concentration in finance, and have run the business side of the company since then.

What aspects of his leadership philosophy have you maintained, and what have you changed?

My father put great emphasis on customer service. He knew that if you kept customers happy, they would return.

For a small business, returning customers and happy staff are key.

I have tried to make sure staff and customers feel they are being heard, especially if they are having a problem or are frustrated. I don’t necessarily subscribe to the “the customer is always right” approach. Sometimes customers don’t know what they need and they are not always right, but as long as we listen to each other, we can work together in a positive way.

DAUGHTER AND FATHER. Caroline Fahmy with her father William A. Gilmore, who founded Educational Data Systems in 1974.

My father (pictured) was also transparent and had integrity in his dealings. An adage he believed in was, “Mean what you say and say what you mean.”

Following his example, I am honest and as open as I can be with staff, customers, and vendors. I have had people tell me that I’m too honest and too open; for me, there is just no other way of working. It helps me sleep at night.

He was an eternal optimist and a big idea thinker. I, too, am an optimist—just ask my staff—but I am more realistic than he was.

When I get big ideas, I write them down and think on them, plant seeds with other people, and then make a decision. My father tended to fly solo with regard to decisions, but I am a consensus builder. I can pull out the “boss” card and make decisions, but I prefer to gather ideas and information from a variety of people, investigate and do research, then discuss options and make decisions based on consensus.

Different perspectives and opinions are important and can make solutions and ideas better. Of course, I surround myself with phenomenal thinkers and I value their input—that’s also key.

You’ve been around for decades as a company, but a lot has changed technologically in just the past five years. What are some of the biggest changes that you’ve had to stay out in front of, to remain on the leading edge? How have they influenced or impacted your work?

Our company has always used high tech in our services and software, starting with early computerized test scoring and early geographic information systems (GIS) technology for school district demographic planning that replaced manual pins on wall maps to represent students.

One of the biggest technological changes in the assessment world is the transition from paper-based to online student assessment. There are still assessment programs that use paper tests and a portion of most assessments use paper for certain groups of students. But almost all assessment companies have gone digital.

We made the strategic decision about a decade ago to remain in paper as long as we could keep our machines running—and truth be told, we would not be disappointed if we are the last company in the country offering paper processing of assessments.

Our challenge is remaining responsive to the needs and wants of state departments of education and school districts as they move in the direction of online assessment.

Our strategy has been to build our own high-tech solutions and build strong partnerships with a small number of very capable companies that provide online assessment services and that need paper processing capabilities.

Sometimes I think the more difficult challenge is not jumping into a technology for technology’s sake. It has to make business sense, but also make sense for education and students.

What role do you see your company playing in helping teachers teach better and students learn better?

Our job as an assessment management company is to make sure the data and tests we produce are accurate, reliable, and valid. We must stick to strict principals of data integrity, fairness in testing, and using the data for its intended purpose. If we do that, then the people and institutions that receive the data we produce can do their jobs.

DELIVERING THE DATA. Educational Data Systems extracts data from a variety of inputs, compiles that data with the utmost of attention and precision, and then securely delivers that data to their clients. Delivering accurate data to help districts, schools, and students succeed has been their passion since first opening their doors in 1974. “Although we have grown since then, and the education world and technology are very different, our passion is something that has never changed,” says Caroline. As a trusted partner of educational organizations—public and private, large and small—they provide industry-leading care in student assessment, research & evaluation, and school district planning services.

Our work on the planning and GIS side is tangential to learning. Our software addresses growth and/or decline in enrollments, school facilities planning (such as opening and closing schools), school capacities and overcrowding, policies around bussing, inter- and intra-district transfers, ethnic balancing, etc.

If these issues get to a crisis point or are not addressed through a thorough and systematic analysis, then teaching and learning are definitely affected.

Assessment has been a hot area, not without its controversies. As a veteran in this space, what is your perspective on some of the brouhaha?

I think you are referring to two possible brouhaha sources: that students are tested too much and the misconception that the Common Core Standards—which early on got an unfair bad rap—are an assessment.

I agree that if teachers do not have enough time in the day to teach because they often must assess students for one purpose or another, that’s not a good thing. Many of my friends and family members (including my mother and father) are California teachers. We often talk about their day-to-day challenges in the classroom, and that the time spent on assessing students is among their biggest challenges.

I will always remember, though, my father saying that it is important to know if what you’re doing is effective or not, and in education, assessment is the way you do that. Assessment, if done correctly, is an objective measurement of the effectiveness of an educational effort.

“I will always remember, though, my father saying that it is important to know if what you’re doing is effective or not, and in education, assessment is the way you do that. Assessment, if done correctly, is an objective measurement of the effectiveness of an educational effort.”

—

In medicine, the patient recovers or doesn’t. In sports, you win or you lose. In education, whether your effort has been effective isn’t that clear and it isn’t always that immediate. Students can pass grade after grade and even graduate, but does that mean they have met learning standards? And even if they have met learning standards, it doesn’t mean they will be successful in life or their careers.

The bottom line is that assessment is a very important part of the educational process. Assessment helps provide feedback to teachers, administrators, lawmakers, and other stakeholders. With assessment results, these stakeholders can better understand how students are doing related to goals and then have the opportunity to make necessary changes along the way to help make improvements.

If there is indeed too much testing, then it is important to cut out unnecessary assessments (which first requires figuring out the definition of necessary), try to reduce or combine assessments, and/or embed assessments into everyday activities and learning so that it becomes transparent to students and doesn’t take away from learning time.

This, then, becomes the challenge for assessment vendors and experts, state departments of education, and school district administrators to make that happen, rather than continuing to pile on new discreet assessments that meet each of their needs. In other words, we need to be smart about classroom time and think twice about using students as guinea pigs for everyone else’s needs.

The second source of the assessment brouhaha is that for some odd reason people began to think the Common Core Standards are not a good thing, or that their adoption leads to more or unfair testing of students, or that the Common Core is a particular assessment. None of these things is true.

The educational process needs standards. The consortium of states, educators, and experts that came together to develop the Common Core Standards were very capable and knowledgeable, and they came up with an amazing set of standards that educators can use to help move learning forward.

As the brouhaha around Common Core Standards began to boil, I began to wonder whether people knew that states have had standards for decades and decades. They are nothing new; what was new was that the standards were updated and relevant to today’s world and were common across multiple states.

Many states decided to customize them, which is also a good thing.

Are they perfect for all states and districts and students? No.

But, the Common Core Standards are as good as or better than educational standards have been in the past.

I think this is a case where knowing your history is important.

There is some movement toward abandoning testing, or at least significantly reducing its role, for a greater emphasis on portfolios and other evidence of student production. Your thoughts on this? Can testing co-exist harmoniously with student portfolios? Or are they inherently at odds?

Yes, testing and portfolios can co-exist harmoniously.

There are many types of tests used for different purposes each associated with different stakes. Some examples are:

• End-of-unit quizzes or in-class assessments used to check student learning of a narrow set of concepts. These are generally low stakes and for formative learning purposes.
• Performance task assessments that allow teachers to see what student can do and know and also where students may have gaps in understanding. Many times, performance task assessments are used to help teachers better learn their content and teaching, collaborate on teaching the content, and/or to reteach or reengage students in the learning process. Performance tasks can be used in low-stakes formative or higher-stakes summative assessments.
• District benchmark assessments given throughout the year to check on student progress toward district goals and learning standards. In some cases, benchmarks are used for higher-stakes decisions, such as moving on to another grade level or graduation. Many times, benchmarks are used to look inward and make improvements to curriculum, teaching and professional development, and/or the assessment itself.
• Statewide summative assessments given to measure student and school and district progress toward goals, accountability, and funding. These are generally high-stakes to the educational organizations from the standpoint of incurring consequences for not meeting goals. Sometimes schools or districts will not receiving funding or have forced interventions on behalf of the district by the state.

Portfolios may be the best form, however, of demonstrating the breadth of a student’s ability, interests, and depth of knowledge.

For example, a student who is interested in art may begin to develop an art portfolio very early in life. As he or she continues to learn and develop skills, the examples in the portfolio will reflect that growth and learning. If that artist is also required to do a science project, and an economics simulation, and a history paper, and a short story, and a film—in addition to taking the state assessment, the physical education assessment, and the English language development assessment, then the student’s portfolio will provide a good picture of what that student is all about.

The thing about assessments is that they are generally designed for narrow purposes and really should not be used for purposes other than those for which they are designed.

But, multiple assessment results combined with a portfolio of student work can produce a more complete picture of a student’s capabilities.

What are some of the lessons you have learned from your experience with the CELDT?

The California English Language Development Test, or CELDT, is a very large statewide assessment program. Since 2000, the CELDT has been used to assess about 1.5 million students each year from kindergarten through grade 12. It is a complex test that involves assessing the listening, speaking, reading, and writing abilities of students, and includes multiple choice and rater scoring of 11 million constructed-response items annually.

I have learned so many personal lessons over the last 10 years of managing this project, but my main takeaway is that the tens of thousands of educators involved with this program diligently work to implement it faithfully.

From the California Department of Education staff and managers, to district testing coordinators, to IT and English language development specialists, to test examiners and trainers, they have all have put forth a huge effort on behalf of English learners in our state. And they do it to help these students succeed.

Yours is a company of measures, statistics, metrics, numerical data, and numbers. To you, what are some of the most interesting numbers in education and why?

The field of psychometrics is fascinating.

For example, it is interesting to me that people can read students’ written responses to a question, evaluate the responses against a scoring rubric, and come up with the same scores, repeatedly.

We produce a statistic that quantifies this called inter-rater reliability (IRR). When the IRR is perfect, it means that the scores of two or more raters agree on the score 100 percent of the time; when the IRR is low, raters less often agree on the score.

For scoring open-ended, constructed-response, or performance task items, teaching people to evaluate and score them with a high degree of reliability is crucial to the assessment process.

Another interesting aspect of educational numbers are statistics that we obtain about the test questions themselves.

These statistics yield information such as the quality of the items (i.e., are they measuring what they were intended to measure over time), whether two tests can be considered equated (and thus interchangeable, like two forms of the same test), and whether a test is multidimensional (i.e., testing two or more constructs, for example, a mathematics test that tests not only mathematics knowledge, but also a student’s reading and writing ability).

The science of educational measurement is fascinating.

What can districts and edtech companies do to make assessments less time-consuming and more useful for educators?

What may be more useful for educators is to bring data into the classroom not as data, but as ideas and solutions to teachers’ individual problems and questions—interpreting the data and helping it tell a story.

It is the difference between, for example, taking someone to a grocery store and saying, “go get what you need”—and giving him/her a recipe for chicken paella they want to cook, along with the saffron, paprika, Arborio rice, and a special pan to cook it in.

“What may be more useful for educators is to bring data into the classroom not as data, but as ideas and solutions to teachers’ individual problems and questions—interpreting the data and helping it tell a story.”

—

Remove the need to walk down aisles, read labels and ingredients, find recipes, buy the specific groceries, and get the kitchen tools to make the dish.

I’m not saying hand teachers everything pre-made—but I do feel teachers’ time is best spent on their specialty: teaching.

Other specialists and administrators need to listen to teachers and help integrate solutions to meet their needs.

What is your approach to student privacy issues, specifically student data privacy? Signing a pledge is one thing. What real words do you have to say about this area? Something that might resonate with a concerned parent, concerned education leaders, or concerned policy makers? What leadership in this area might you provide to a sector that might steamroll any thoughtful approach in the name of business?

Protecting student data must be in the forefront of all education stakeholders’ thinking and work. One big issue I see is electronic transfer of student data.

Once data is transferred electronically, it’s difficult to keep control.

Copies of data may be in multiple places and all of a sudden, it no longer is protected.

For example, if an educator attaches a spreadsheet containing student names, IDs, and test scores and sends it in an email, that data is now in his “sent” box, the work inbox of the three people he copied, all of their smartphones, laptops, and iPads when their email pops in, and in their Gmail or Yahoo accounts.

All of that private student information may be in dozens of places, some of which could be accessible to other children who might use the smartphone to play a game or spouses or friends who borrow a laptop or iPad. We must all be aware of the potential dangers and exposure of emailing student data. 

Our approach to this particular issue is to develop custom data transfer protocols for our projects that use secure encryption and keep the data as close to the source as possible and only accessible to the people who need to use it. 

If anyone sends us data in an email or if we discover any other type of data breach, we immediately notify them of the issue and delete the data file.

The data we produce belongs to our district clients. We would never think of using it for our own purposes or for mining or advertising.

As paper-based assessments get rarer, how do you see your solutions evolving in the next five to 10 years?

In our assessment work, we offer solutions that combine paper-based assessments with high-tech solutions. We have a paper-based assessment software-as-a-service (Saas) for classroom-based performance task assessments. Although students take these assessments on paper, the SaaS provides computerized student records, pre-printing of tests, local scanning, and online training and scoring for teachers and raters to read and mark scores.

Reports are online and on demand. We also have a suite of online tools to help test coordinators manage their assessments.

This suite offers electronic pre-identification of student tests; student demographic data correction (to help ensure that reports are as accurate as possible); online request for a pickup of test materials; and online archiving of student test results along with downloadable data files for integration into student information systems.

We have used these tools for the last few decades to help make paper-based assessment programs more efficient and more accurate.

Other changes that we have continued to keep up with are in GIS and mapping technology. School districts’ needs are a function of their geography and student demographics, and so GIS plays a role in the planning and management of district facilities, student population changes, attendance zone planning, etc.

We have continued to upgrade and build new software that utilizes the newest GIS technology. We offer a SaaS, LocateMySchool, that uses a Google-based API to map school attendance zones, look up addresses and show important points within the district (such as bus stops and event locations). We are also presently upgrading our flagship software product ONPASS Pro to utilize the latest ESRI platform ArcGIS Pro. Electronic data transfer, automatic linking of data systems through APIs, and servers on the cloud are a standard part of our work as well.

Now, for a very broad question: What is the state of education today?

I am not knowledgeable about the whole of education, but when I talk to the teachers I know, they are struggling.

They have larger class groups and fewer resources.

Their schools have a lack of basic services like air conditioning and heating.

They have more to do every day, and much of what they have to do has nothing to do with teaching students.

Teachers are in the trenches of education and on the front lines of kids’ lives.

As we have seen, they have kids’ safety and lives in their hands.

Personally, I feel that the money that seems to have been sucked out of our society and into the hands of a few needs to make its way back to the classroom.

Back to the teachers and principals and educators who are responsible for the one-on-one education of our kids. It’s not just about money, but that’s a big part of it. We need to give teachers more resources, more support, and more respect as professionals. 

DATA TO LIVE BY. Caroline Fahmy in 2018. Says Caroline, “The less we interact with our fellow human beings and the more with computers, the more we lose the richness and subtleties of life, from little things like seeing a raised eyebrow during a conversation to getting a hug and smile to just the feelings of love and support and happiness from each other.”

On the other hand, I also see a thriving, busy, and hardworking industry that despite very difficult conditions is doing an outstanding job.

From teachers to school nurses to principals to district office personnel and counselors, educators are amazing and dedicated to the students they serve. 

What do you believe is technology’s role in education? 

I feel technology’s role is the same in education as in other industries: as a tool to make life better for humans. 

What makes you say that?

I recently listened to an interview on NPR with three top artificial intelligence experts at Google, Microsoft, and Facebook. An interesting point they made—and all three adamantly agreed—is that AI’s purpose is not to replace humans.

AI is a tool, like paper and pencil are tools, to extend human capabilities, ultimately for the betterment of humanity. I hope this is true about AI in particular and about all technology.

I often reflect with my millennial children what life was like without computer technology—I grew up in the 1960s and 70s.

Although I am thoroughly entrenched in technology both at home and at work, our conclusion is that while high tech may be important to our daily lives, and even fun sometimes, people need people, especially in education.

The less we interact with our fellow human beings and the more with computers, the more we lose the richness and subtleties of life, from little things like seeing a raised eyebrow during a conversation to getting a hug and smile to just the feelings of love and support and happiness from each other.

Technology is very powerful and has almost infinite potential, but I hope we can use computers and technology to supplement and make life better, never to replace what’s most important in life: human relationships. It takes a concerted effort and smart decisions to not let technology take over our human relationships.

“It takes a concerted effort and smart decisions to not let technology take over our human relationships.”

—

Not too far into the future, in a more virtual reality/augmented reality kind of education world, how will assessment fit in? Is your company looking ahead to any particular future trends and beginning to do any behind-the-scenes movements to accommodate, respond to, or even shape such areas?

As an example of what might be possible, I look at the ability of computers to track the keystrokes of a student during online testing to better understand their thought processes. I think in a virtual reality world of learning, computers will be tasked with assessing students as they interact with the computer. The assessment could be built in to the lessons.

With online assessments, items are enhanced by the computer.

So, instead of the static question and answer we’ve traditionally seen with paper/pencil assessments, computers can be more interactive with students as they work through problems and give their answers. Static items have turned into technology-enhanced items, which may turn into virtual-reality-enhanced items in the distant future.

Anything else you care to add or emphasize about edtech, assessment technology, education, or any other area you’d like to comment on?

At one time, pencil and paper technologies transformed education. We are part of a transformation brought about by digital technologies, and it is an exciting time to be in the field of educational assessment.

—

Victor Rivero is the Editor-in-Chief of EdTech Digest. Write to: victor@edtechdigest.com