BASELINE PILOT STUDY:
Information Seeking Habits – Pilot Findings
December 2011
Background overview
We decided to take two groups of learners that have been described by teachers as ‘worlds apart’ in the way they apply and digest information and conduct an experiment to test the validity and reliability of student confidences in using and applying technology survey (Baseline Case Study B). Our initial findings in the early days of the student survey (Baseline Case Study B) provided us with information that suggested our students had a greater confidence in using technology than teachers were reporting (Baseline Case Study C). During an initial meeting teachers raised considerable concern that although students felt they were confident finding and using information, especially using Google, these skills were in fact much weaker than students themselves perceived. In fact teaching staff felt that students had grossly overestimated their competence when finding and using information academically.
With this in mind we felt this project was the ideal opportunity to scrutinise the differences of opinion. As a result during our baseline process we have conducted and piloted a small scale hands-on activity task to measure the reality of such confidences using a series of task based activities to investigate and observe the reality in process.
The task was conducted with two categorised groups from an A-level profile and from a level 3 FE vocational profile to identify similarities within the age group and differences in their educational background and profile. This was also conducted with the intention to pilot a wider investigation for latter in the project.
The tasks were all conducted in one study centre to ensure the consistency of environment and equality of resources available (Fig 1.1).
Fig 1.1: Study Centre environment used for the pilot study
Overview of Methods
Students were asked to seek three information sources based upon common concerns raised by teachers. These were that learners did not truly understand the use of Google or interrogate answers in a scholarly manner. They claimed the students relied too heavily on Google to do the thinking for them and relied on Google for everything even when other more reliable resources were close at hand. Questions were created to measure the truth about this. The first question asked participants to look for information in a book, the other two were asking for specific information that could be either found on the web or in library based archives. The latter two were based on actual prior experiences they had encountered with students when asking them to do such tasks.
Results
The results from each of the three questions piloted are described in turn
Task One Question: Book Task
In the book Business L2 BTEC First (Pearson 2010) there is a quote on a yellow background in the chapter of financial forecasting asking “What is so special about the cash flows of seasonal businesses” on which page is it on and what is the image it is linked to?
Fig 1.2: Table of results for book task
|
|
FE Level 3 Vocational |
% of 28 students |
AS & A2 A-level |
% of 25 students |
|
Went for the book on the shelves |
17 |
60.7% |
23 |
92% |
|
Went on Google/Amazon/Other |
11 |
39.3% |
1 |
4% |
|
Asked in Study Centre for directions |
3 |
10.7% |
10 |
40% |
|
Found the correct answer |
12 |
42.8% |
23 |
92% |
A very noticeable difference between the two groups was their use of books. The A-Level students went for the book on the shelves which correlated with them finding the right answer. Interestingly there seemed to be a much wider use of Google to find the book from the FE Vocational group. I found this interesting as it did seem to uphold the teachers’ view that these two groups sought and consumed information differently from different sources.
It is interesting that although these students are in the same age bracket yet their habits for seeking information are different. It does demonstrate that learners in different subject areas or qualification structures have different information seeking styles therefore do not necessarily conform to the stereotypical notion that all learners turn to the web or that all learners have a preferred style of learning and finding information.
Google Predicted Tasks
The next two tasks were based around the predicted use of Google and student competence when finding, using and evaluating information found on the web or from online sources. For this we will be comparing basic data from each task (Fig 1.4 and Fig 1.5) with basic data gained in this area from Baseline Case Study B (Fig 1.3).
We surveyed students to find out how confident/proficient they believe themselves to be in finding and using information. Then we got them to actually do some associated tasks to see if their perceptions matched the reality of skills.
Fig 1.3: Table of results from Baseline Case Study B
The confidence levels were very high for their perceived proficiency on using search terms in Google to find information, yet using the more formal ways to seek information was no so apparent.
We wanted to see if this was the case in reality. By their own admission they should be good in finding the information on Google rather than anywhere else.
We then set students some tasks to find information online. We chose tasks to find a specific date or a specific event and also asked them to find a front page newspaper headline.
Task Two Question: Daily Play Lottery Task - Google Reliance
In what year was the daily play lottery started?
Fig 1.4: Table of results for lottery task
|
|
FE Level 3 Vocational |
% of 28 students |
AS & A2 A-level |
% of 25 students |
|
Exact question into Google (copy and paste) |
24 |
85.7% |
11 |
44% |
|
National Lottery Site Used |
3 |
10.7% |
4 |
16% |
|
Answered with national lottery date - incorrect |
23 |
82.1% |
14 |
56% |
|
Answered with daily play lottery date - correct |
2 |
7.1% |
9 |
36% |
An alarming finding is that those who used Google mainly did using copy and paste. This was interesting as they copied this directly from the paper worksheet, suggesting possibly that copy and paste is not just confined to computer use. An alarming amount of people got the answer wrong and instead of answering with the daily play lottery date they answered with the national lottery date (the vast majority got their answer from wikipedia as the top answer in the search engine). It can only be assumed that as this is the answer Google offered in the first page of results thrown back in the search this indicates a very shallow analysis of the results or process of finding and sorting through infornation. This does correlate with the survey that overall less than 40% felt they were good at evaluating whether a source was valid reliable or trustworthy. It seems that the first answer they came across was the answer they used without checking it was correct. This is rather worrying.
The two processes involved in finding information on Google are first the search terms (which in majority was copied and pasted) and then sourcing our a reliable or correct answer which is alarmingly low, most especially with the FE Vocational group at 7.1%
There is definitely a strong need to support information seeking skills from the entering of search terms right the way through to evaluating the sources they find, even students themselves perceive their use of Google as proficient.
Task Three Question: Newspaper Task – Google Reliance
What was the front page headline in the Guardian Newspaper on Sept 25th 2003?
If you type this question into Google the top answer gives you the incorrect headline from Sept 23rd 2003, not Sept 25th 2003. We wanted to see if copy and paste meant reliance on Google for the right answer, and Google only without referring to other sources or checking.
Fig 1.5: Table of results for newspaper task
|
|
FE Level 3 Vocational |
% of 28 students |
AS & A2 A-level |
% of 25 students |
|
Exact question into Google (copy and paste) |
14 |
50% |
4 |
16% |
|
Guardian site searched |
3 |
10.7% |
13 |
52% |
|
Newspaper archive searched |
5 |
17.8% |
7 |
28% |
|
Answered with Bruno headline (wrong date) |
11 |
39.2% |
1 |
4% |
This offers another worrying use of shallow copy and paste search techniques that seems to correlate with wrong answers. What is most interesting is those that searched using the more formal methods such as the guardian site or newspaper archives correlates with less incorrect answers. However to search these types of resources is not necessarily a shallow task that can be done with copy and paste.
Discussion of the Findings
These are initial findings only and should be treated as such, they are very basic comparisons. These findings are only comparisons between level 3 students Vocational Versus A-Level just out of sheer curiosity of like for like students following different academic/vocational study paths.
Other elements noticed between the two comparable sets of students during observation were their behaviour, ability to focus and engagement with the tasks they were asked to perform.
Most worrying is the shallow depth of searching, the use of copy and paste and the inability when using to Google to source the correct or most suitable answer. There is a distinct difference in the results between those who used more depth of searching and those who copied and pasted. However the amount of shallow search techniques was rather alarming.
The two groups performed very different as predicted by the teachers and goes to show that not all youngsters today have the same characteristics. This seems to depend on their programme of study and how that affects their use and sources of information they use.
These differences are only two groups, yet it indicates the needs for a wider investigation of the different types and levels of learners within the college.
Deeper investigation
These figures will be formally scrutinised alongside all groups across the college for the purpose of the Digital Literacy Project using more formal methods.
From this we intend to widen the activity to include finding video resources and asking participants to look for more scholarly and credible content or information. We also intend to further analyse the comparative data between the perceptions of confidence survey (Case Study B) and findings from the hands-on tasks.