Accessions to the DL shall be governed by a formal written collections policy, approved and adopted by the community, which ensures that all content is relevant to Earth system education and meets minimum standards as implemented by DL staff. Within the distributed DL collection, there shall be a sub-collection comprising high quality teaching and learning materials which have been rigorously reviewed and evaluated according to published selection criteria.

The DL shall collect materials that facilitate learning about the Earth system. The collection shall favor learning resources that bring the Earth into the classroom, and connect the general with the specific, theory with evidence, and the global with the local. The scope of the collection is Earth system education, with particular emphasis on interdisciplinary areas. Materials are selected to support education by giving both educators and students access to tools and resources for instruction and research. Initial priority in the building of the collection shall be given to materials for undergraduate education.

Types of materials to be included in the collection:

• data,

• tools to look at/analyze/manipulate data,

• methods and agents,

• text (scientific papers, summaries, indices and abstracts),

• images,

• models and simulations,

• animations and videos,

• lesson plans and curricula, including assessment tools,

• learning activities (guided inquiry sets, modules, problem sets, classroom and laboratory activities),

• educational multimedia,

• tutorials about the Earth for students,

• interactive mini-tutorials about pedagogy, for faculty, grounded in learning science research, targeting specific pedagogical problems (e.g. students' difficulty with 3-dimensional structures, or maps, or geological time)

• virtual field trips,

• student portfolios (examples of student work).

The DL will accept materials in a wide range of digital formats and use technology to overcome format differences. The DL may offer guidance to providers regarding formats and templates.

1. Accuracy, as evaluated by scientists

2. Importance/significance

3. Pedagogical effectiveness. Is there evidence that student learning has occurred?

4. Well documented. Data shall have metadata, lessons shall have rubrics, etc.

5. Ease of use for students and faculty

6. Inspirational or motivational for students

7. Robustness/sustainability

 

Users' perspective:

Students and educators are constantly in search of quality teaching and learning materials. They don’t necessarily have the background to evaluate those materials themselves, and they don’t have time to experiment with a large number of resources. The reviewed section of the DL will be a recognized, efficient source for quality teaching and learning materials.

Creators' perspective:

Creators of educational resources are not currently rewarded in most cases for their creative effort. Inclusion in the peer-reviewed section of the DL will come to be a recognized stamp of professional approval at the level of publication in a peer-reviewed journal.

At the present time there is very little out there that will meet our selection criteria if applied rigorously. Users are seeking materials on a huge range of topics. The DL provides added value by being inclusive while providing powerful search and classification capability. The library will identify these resources as relevant but unreviewed, applying the philosophy of caveat emptor.

The selection criteria shall be applied by the DL staff informally for initial inclusion in the library. The DL staff shall include expertise in Earth systems and experience in education, as well as in computing and librarianship.

A formal review process involving the peer community will apply these same criteria more rigorously for inclusion in the reviewed sub-collection. We envision a process analogous to a journal peer review process, with an editor-in-chief, associate editors, and volunteer reviewers. To evaluate the submission against the full suite of selection criteria, the reviewers and editors might receive several intertwined (hyperlinked) documents to review, instead of one manuscript. For example: (1) An assignment for a learning activity, as it would be presented to the students; (2) a practical document about how to teach this activity, written for the faculty member, including answers, scoring rubrics, common problems in teaching this topic with suggested solutions, pointers to related resource, etc.; (3) a document describing field testing of the activity and providing evidence that student learning has occurred.

It will be the ultimate responsibility of the creator to document the pedagogical effectiveness of material submitted for inclusion in the peer-reviewed core collection, but the DL will facilitate this process. The DL itself can serve as a 'testbed' for reviewing and field-testing educational materials, and positive feedback via communication networks hosted by the DL could provide the basis for the documentation of educational effectiveness required for an item to move into the peer-reviewed section of the DL. The DL might choose to develop new assessment mechanisms to evaluate the effectiveness of innovative types of educational materials. These assessment instruments would then become the property of the DL, and could be more broadly used in testing other educational materials. And, indirectly, we hope that the well-publicized existence of the "pedagogical effectiveness" selection criterion will encourage colleges and universities to establish teaching resource centers, staffed with people skilled in evaluation, who will be paid by the college/university to evaluate the effectiveness of the educational materials being developed and used on their campuses.

The dynamic nature of the DL allows the review process to continue past the initial submission and acceptance. Correspondents can report on ways in which the materials were adopted or adapted to meet special circumstances, comment on the effectiveness of the activity, or suggest ways to more effectively use the activities. This continued commentary may be attached to the original submission; thus the original contribution will continue to gain value as experience demonstrates how to best use the materials.

Action items

1. Circulate the draft collection policy and criteria for acceptance to potential users for comments.

The section of this report dealing with collection and selection policies should be circulated to a broadly based community of potential users and contributors. A committee should be formed to manage the consultation process, synthesize results, and prepare responses and changes for review by DL leadership. This process should be completed, and formal policies adopted, within six months.

2. Define what 'well documented' means for each category of materials.

This will require extensive consultation with the group(s) dealing with classification, metadata, documentation, and discovery. The minimal criteria for completeness of documentation must be established, such that items which fail to meet these criteria are not accepted into the library. This task should be carried out by the group(s) responsible for documentation, and completed within one year.

3. Draft guidelines for what 'pedagogical effectiveness' means.

In collaboration with the group working on item (1) above, a group should determine what methods will be used to document and judge pedagogical effectiveness, and should develop specifications of minimum standards, recognizing that different methods and standards may be appropriate for different kinds of submissions. This task should be completed within one year.

4. Research recommendation engines and methods for review and evaluation.

Many methods might be used to review and evaluate items in the DL collection and to deliver results to potential users. The geoscience community is familiar with peer review, by which referees comment on material, often anonymously. Educators are familiar with formative and summative review of material, with a strong tradition of empirical (field) testing in classrooms. Digital librarians are familiar with review mechanisms common on the WWW, including unsolicited reviews submitted by individuals (e.g. amazon.com) and WWW-based surveys. The DL should initiate a review of methods, representation of results (e.g. in metadata), and associated software for elicitation of reviews and their support within the library. Legal implications (libel) should be researched. This work should be completed within two years, by a committee of volunteers, and presented in the form of a series of recommendations to DL.

5. Establish mechanisms for providing academic career credit to creators/suppliers of materials.

Creators of materials that are accepted into the reviewed collection should receive credit for their creative activities comparable to the academic career credit received for publication in refereed journals. A committee should be formed to develop a plan to achieve this objective through appropriate consultation and collaboration with institutions nationwide. It should report within 12 months.

6. Develop scenarios for DL use, and coordinate them with collection development.

Good strategies for collection development can arise from well-crafted scenarios for DL use, particularly during the early phases of development. A group should be assigned the task of developing scenarios of use, and scenarios for the supply of materials (see the scenario in this report for initial guidance), and should abstract collection requirements from them for consideration by DL leadership. This task would be useful during the first two years of DL development, and should be scheduled accordingly.

7. Develop specifications for prototype collections.

Specifications should be developed for 1-, 2-, and 5-year milestones, and should include the full breadth of materials anticipated for DL. A group should be formed to prioritize collection-building, and to specify the milestones to be achieved by the collection at three milestones. This task should be completed within six months. This group should evolve into a permanent group charged with monitoring and maintaining the collection, with the power to implement and to recommend changes in all aspects of the collections policy, and to solicit and discontinue elements of the collection.

Example User Scenario

Setting:

Faculty Lounge in a University that is not far, far away.

Brad looked at Felicia skeptically, "You're telling me to teach global climate change in a geology class AND make it relevant to a bunch of college sophomores. That's a pretty tall order!"

Felicia, ever the optimist replied, "I suppose. Let's check out that new Digital Library. I heard it was supposed to help us connect the global to the local. "

Sitting down at a nearby PC, connected through high-speed Internet-2 connections, the pair of geology professors quickly moved to the DL's search interface. Using the map interface they zoomed into a hundred mile region around their campus, and set the topic index to global climate change. Topping the list that the search engine returned was a climate record from Lake Mendota going back in geologic time, including warm periods when the Earth's poles were not topped by polar caps.

"Hey," Brad called out, we can use the past history of Lake Mendota to fuel scenarios for the future. We'll have the students look at past climate and CO2 levels to predict what might happen in future climates."

"Sure…" Felicia ironically replied "we'll substitute the difficulty of understanding large-scale space of the earth with the difficulty of understanding deep time and how climate and the atmosphere have varied over time in one location. That sounds a lot easier."

"No, wait, here's something that might help. It seems like a mini-tutorial for faculty about helping students understand processes on geological time scales. Look there's some guy here who actually investigates how people develop their understanding of time as his research topic, and here are some really practical suggestions about overcoming the problems students have with 'deep time.'" Brad pauses. "Well do you have anything better to suggest?"

"Who me?" Felicia replied innocently. "Well, there's always policy implications. Fact is these kids are going be inheriting this environment, so they get real excited about looking at the implications of science for society ."

Going back to the search engine they again ask for the global warming theme, but asked this time for future scenarios. This search led to several hundred results, but luckily the search engine sorted them into three categories of "Models & Supporting Data," "Severe Weather," and "Agricultural Impacts," and ranked the returns in each category. "Models" included the models and supporting data used by the Intergovernmental Panel on Climate Change, including atmospheric carbon dioxide data from Mauna Loa and global temperature and precipitation data sets data; java applets to visualize the data; and several precalculated scenarios showing the effects of a doubling of CO2 over a century period. "Severe Weather" included a 3D model of atmospheric turbulence, and "Agricultural Impacts" included maps of predicted changes in crop yields.

"Hmmm… We don't have this exactly local, but we may have a rich enough set of pieces so that they can balance their ways of being remote and local" Felicia mused.

"You've lost me. It looks like way too sophisticated for the kids in our classes. Are you sure you don't want to do it with the textbook like last year?" Brad urged.

"Hold on. Let's give this a chance. Say we start with the changes in global carbon levels as indicated at Mauna Loa and local recording stations to analyze change over time, looking at global trends and local differences. This establishes the primary circumstantial evidence for global warming. Then we'll move on to the Lake Mendota data to study past eras of increased greenhouse effect warming. Finally we bring it back today by having groups of students do modeling using these scaled-down versions of the IPCC models looking at both global and regional effects. This should be a pretty substantial project involving the students in several weeks of work. At the end, they should present their findings to one another. These presentations set the scene for them to analyze differences between the models, and between models and data, and think about how all these different models make policy decisions difficult."

"Well you're the department chair, so I guess that must be the way to do it." Brad glumly replied picturing his weekends for the next semester melting away with the work of preparing activities, reading student papers, and learning more about global warming than his specialty demanded.

"Well you don't have to be so long faced about it. Remember when you're up for tenure review this will look awfully good. These activities and curricula you're developing can be registered in the Digital Library and count as peer-reviewed publications in your review. BUT you have to collect data on student learning. Maybe we should to talk to one of the faculty in education?"

"Yeah, this could be pretty cool" Brad exclaimed, finally getting caught on the excitement of it. "But the school of education is all the way on the other side of campus." Then catching the look on Felicia's face he went on, "Well, I guess I could talk to someone, sure."