In the summer of 2022, eScholar released version 2022 of the eScholar CDW. This release comes with a major enhancement to improve performance. Customers that upgrade to version 2022 will experience ETL vNext, the eScholar CDW’s new ETL engine. The conversion to ETL vNext began with version 2021 and is now complete with the release of version 2022.
Clients who have upgraded to version 2022, are already experiencing significant performance improvements.
For example, eScholar analyzed the ETL load times for the Pennsylvania Department of Education (PDE), which is one of eScholar’s largest customers. PDE uses the eScholar CDW for its statewide K12 longitudinal data system, PIMS. As part of this analysis, eScholar reviewed the ETL load times for version 2021 and version 2022, specifically for four ETL load plans: student snapshot, student, school enrollment, and program. Every October, local education agencies (LEAs) submit this data directly to PIMS, which is foundational for data collection for the rest of the year.
The time savings between version 2022 and version 2021 are significant.
For a person responsible for loading the data, version 2022 was four times faster than v2021. The average time to complete the execution of four load plans was 242 seconds or about four minutes. With v2022, the average time to complete the four plans was 68 seconds or about 1 minute.
Expanding this throughout the state, the upgrade to version 2022 saved over 195 hours of time across the state. That’s over eight days of time saved.
In eScholar CDW v2022, we completely overhauled our legacy ETL Engine. With this conversion eScholar upgraded the embedded ETL Engine from a commercial product to In-House built ETL Engine built on the Java platform. This upgrade of the ETL Engine gave eScholar clients a performance gain of over 60%. This performance boost was verified and tested in Pennsylvania’s eScholar CDW in October of 2022.
To put perspective on these significant changes, we put together this sample of the testing data from 2021:
| PLAN NAME | 2021 | 2022 | Percentage Improvement |
| STUD_SNAPSHOT | 117 | 21 | 82% |
| STUDENT | 93 | 36 | 61% |
| SCHOOL_ENROLL | 14 | 5 | 64% |
| PROGRAMS_FACT | 18 | 6 | 67% |
As we mentioned above, our ETL vNext engine is a custom, in-house-built engine that gives our customers massive performance gains regardless of if they are on an AWS Server or if they are self-hosted. Even though self-hosted customers cannot take advantage of the scalability of AWS Aurora PostgreSQL, the ETL engine upgrade provided them with over a 60% performance gain on their ETL processing. These significant changes give us the flexibility, speed, and reliability needed to adapt and keep present in the ever-evolving technology and data models.
It is a high priority for us here at eScholar to ensure that we are giving our customers and future customers the performance and reliability that our customers need to process the heavy data loads in times such as today. With over 20 years of experience in the education data field, we have seen incredible and significant changes to data, data reporting, data visualizations, and data processing. This experience has given us the insight and the foresight to be ready and prepared for data’s fast-evolving technological advances.
