The
ICT Literacy Assessment was designed to test students’ abilities to u=
se
technology to solve information problems, to think critically about
information, and to understand and comply with legal and ethical limits.
Assessment tasks ask students to define, =
access,
evaluate, manage, integrate, create, and communicate information
using simulated software tools. These ICT Literacy proficiencies encompass =
the
skills that help students function in our information-rich society.
The
ICT Literacy Assessment is a large-scale assessment designed to describe the
ICT Literacy skills of a student population or subgroups of that population
(e.g., transfer students, students of a particular race/ethnicity, physical
science majors) with appropriate sampling methods and to support
institutional-level decisions. Rather than producing an individual score re=
port
for each student, the ICT large-scale assessment produces a score report fo=
r an
entire testing sample or for subgroups if institutional sampling supports s=
uch
comparisons. The target population for the ICT large-scale assessment is
students entering upper-level work: students at 4-year institutions about to
enter their junior year and students about to graduate from 2-year institut=
ions
who plan to transfer to a 4-year institution or enter the workforce. Studen=
ts
taking the assessment should:
·
Function in English at the
college level (the language of instruction)
·
Possess basic technical
competence (ability to use a mouse and keyboard)
The
large-scale ICT Literacy Assessment is designed to inform several types of
institutional decisions. This large-scale assessment provides information f=
or
administrative decisions about groups of students, but does not provide sco=
res
for decisions about individual students. Intended uses for the assessment
include:
·
Curriculum planning. The assessment provides a description of the ICT Literacy skills o=
f a
student population. This description can suggest skills that need additional
emphasis in ICT-related courses.
·
Curriculum evaluation. Appropriately comparing overall student perform=
ance
before and after delivery of a particular curriculum can contribute to an
evaluation of the effectiveness of the curriculum.
· =
Accountability data=
. Assessment results appropria=
tely
comparing entry-level students to more senior students can provide informat=
ion
on the success of ICT education programs at an institution. The assessment =
can
also contribute to investigations of improvement in students’ ICT
literacy over time.
· =
Individual decision-making. The assessment does not provi=
de
scores for individuals, so it is inappropriate to use the assessment result=
s to
make decisions about individuals.
· =
Technical certification. The assessment focuses on
problem-solving and critical thinking skills in the context of technology.
However, this is not a technical certification assessment, and no inferences
should be made about test takers’ ability to use specific software
products.
How
to Interpret This Report
Do the test t=
akers
reflect my school’s student body?
If you conducted probability
sampling and have a Score Report that includes standard errors and signific=
ance
tests, your test results may be an accurate reflection of the student body.
Examining Report A1 may reveal whether the demographic =
characteristics
of students tested differs from the demographic characteristics of y=
our
student body.
Report A1 describes the gender
and ethnic composition of students at your school. The first column in the
table describes the gender and ethnic classifications used. The second colu=
mn
describes your school’s overall composition in terms of these
classifications; each cell contains the percentage of students at your scho=
ol
belonging to a particular gender or ethnic group. This information provides=
you
with a very rough baseline for evaluating whether the students tested resem=
ble
your school’s population.
If your school used probability sampling to select stude=
nts, t=
he
third column describes the percentage of students selected for inclusion in
testing belonging to a particular gender or ethnic group. This provides a r=
ough
way for you to evaluate how well the sampling was done. That is, if percent=
ages
in column three roughly correspond to percentages in matched rows in column
two, you can have some confidence that your selection process resulted in a
group of students similar to your student body. The fourth column describes=
the
percentage of tested students belonging to a particular gender or ethnic gr=
oup.
This allows you to evaluate whether participation by gender or ethnic group=
is
consistent with their representation in the selected sample and student bod=
y.
If your school did not use probability
sampling to select students, the third column describes =
the
percentage of tested students belonging to a particular gender or ethnic gr=
oup.
This provides a rough way for you to evaluate whether the demographic
characteristics of test participants are similar to the demographic
characteristics of the student body.
What
is the ICT Literacy Assessment?
The ICT Literacy Assessment is a comprehensive test of
Information and Communications Technology (ICT) proficiency specifically
designed for the higher education environment. It uses scenario-based
assignments to assess ICT skills required of today's higher education
students – not just knowledge of technology, but the ability to use
critical-thinking skills to solve problems within a technological environme=
nt.
Define: The
ability to use ICT tools to identify and represent an information need
appropriately.
Example activities include:
Access: The
ability to collect or retrieve information in digital environments. It incl=
udes
the ability to identify likely digital information sources and to get the
information from those sources.
Example activities include:
Manage: The
ability to apply an existing organization or classification scheme to
reorganize digital information. It includes the ability to identify
pre-existing organization schemes, select appropriate schemes for the curre=
nt
usage, and apply those schemes.
Example activities include:
Integrate=
: The
ability to interpret and represent digital information. It includes the abi=
lity
to use ICT tools to synthesize, summarize, and compare and contrast informa=
tion
from multiple digital sources.
Example activities include:
Evaluate<=
/span>: The
ability to determine the degree to which digital information satisfies the
needs of the task in ICT environments. It includes the ability to judge the
quality, relevance, authority, point of view/bias, currency, coverage, and
accuracy of digital information.
Example activities include:
Create: The
ability to generate information by adapting, applying, designing or inventi=
ng
information in ICT environments.
Example activities include:
Communica= te: The ability to communicate information properly in its context of use for ICT environments. This includes the ability to gear electronic information for a particular audience and to communicate knowledge in the appropriate venue.<= o:p>
Example activities include:
What
do we mean by High, Middle, and Low Performers?
Student performance is grouped on the basis of relative performance.
For ICT, we created three performance groups – high, middle, and low
– for overall performance and for each content area. These groups are
defined by dividing the total group of all students tested in 2005 across a=
ll
campuses into thirds on the basis of test performance. The content-based
descriptions of these performance levels for each content area are presente=
d in
a subsequent portion of this text. This approach allows for the comparison of performance
across content areas in a way that has a consistent meaning.
How
did my students perform?
Overall Skill.
Report A2 presents the percentages of students at your school whose
performances were at each of the proficiency levels for the overall ICT sco=
re.
Each of the bars represents a proficiency level. Their heights reflect
percentages of your students so that if “High” has a value of 5=
0%,
this means that half of your students performed at the “High”
level.
If your school conducted probability sampling, the standard
errors of each percentage are also listed. The standard errors
represent the degree of uncertainty in the percentage reported based on the
sample of students testing. If the standard error is nearly zero, the
percentage is likely to be very similar if a different group of your studen=
ts
was randomly selected and tested. If a standard error is large, that percen=
tage
is likely to be very different if a different group of your students tested=
.
The leftmost bar of Report B1 presents the mean ICT score, and,
therefore, the performance of a typical student for your school. If your sc=
hool
conducted probability sampling, this value is represented by a horizontal l=
ine
at the score value on the y-axis as well as numerically in the center of the
bar. With appropriate selection of test takers, your school’s average
performance can be interpreted as the level of ICT skills possessed by the
typical student at your school. The size of the bar from top to bottom
corresponds to a range from the mean ICT score plus two standard errors to =
the
mean ICT score minus two standard errors. The standard errors
provide information about the degree of uncertainty associated with the mean
because of the particular sample of students tested at your school. If a
different sample were selected, the mean score would likely fall into this
range of scores.
If your school did not conduct probability sampling, the mean ICT score for your school is indicated by the height of the leftmost bar in Report B1. These outcomes reflect the performance only of tested students.<= o:p>
Possible scores range from 0 to 300 with a mean of 150. A score of =
150
represents the average performance of all students from all schools
participating in the 2005 administration. The standard deviation of these
scores is 35. Scores were calculated for 4,899 students from 31 participati=
ng
schools. Most of these schools did not use probability sampling. The schools
were also not nationally representative. For these reasons, the mean and
standard deviation of scores do not reflect national norms. However, they c=
an
be used to compare the performance of test takers at a school with the perf=
ormance
of all test takers in 2005.
The average ICT score for the 2005 administration is represented by=
a
dashed line in Report B1. If a school has conducted a probability sample, t=
he
bar overlapping with the dashed line at 150 means that you can not be certa=
in
that your students’ mean performance is greater or less than the mean
performance of all test takers. When the bar is entirely above the dashed l=
ine,
the evidence suggests that your school’s mean performance is likely
better than the performance of a typical student across all schools. If your
school did not conduct random sampling, the dashed line provides a point of
reference for your school’s performance. However, it is not possible =
to
evaluate that comparison statistically.
By Subproficiencies.
Report A3 presents the percentages of students at your school whose
performances were within each of three proficiency levels for ICT subproficiencies. Each bar represents a subproficiency and is “stacked” so=
that
each bar totals to 100%. That is, the bottom section of the bar represents =
the
percentage of your students in the “Low” group; the middle sect=
ion
represents the percentage of your students in the “Middle” grou=
p;
and the top section represents the percentage of your students in the
“High” group. If your school conducted probability sampling,
standard errors are included in the table below the chart. A standard error may also be
reported as “NA” should the number of students be too small to
perform the calculation.
Your school’s subproficiency perf=
ormance
is also described in Report B1 where each bar represents the performance of
your students in a subproficiency area. These d=
ata
are similar in meaning and interpretation to the overall proficiency column=
in
the same plot. This column was previously described under ‘Overall
Skill.’
If your school conducted probability sampling, approximate comparis=
ons
of subproficiencies can be conducted by reviewi=
ng the
bars for overlap. The extent to which two bars do not overlap suggests that
performance is different on these subproficiencies.
This comparison may inform the relative priorities of different subproficiency areas in establishing curricular empha=
sis. It is important to note that h=
igh
correlations among subproficiencies may cause v=
alues
to appear different when they are, in fact, not different.
How
did my students perform by subgroup?
If =
you
indicated an interest in comparing the ICT skills of particular groups of
students, these begin with Report C1-1. An example of such a comparison mig=
ht
be called “School Year” with groups labeled freshman, sophomore,
junior, and senior. Each bar represents the percentages of students whose
performances were at each of the proficiency levels for the overall ICT sco=
re
for a group you selected. Each bar is stacked so that it tota=
ls
to 100%. That is, the bottom section of the bar represents the percentage of
your students in the “Low” group; the middle section represents=
the
percentage of your students in the “Middle” group; and the top
section represents the percentage of your students in the “High”
group. As with other reports in this document, standard errors are included=
in
the table below the plot for sch=
ools
that conducted probability sampling. If a school conducted probabili=
ty
sampling, bars are not presented for any group with fewer than 50 students.
Standard errors are reported as “NA” for such groups. A standard
error may also be reported as “NA” should the number of student=
s be
too small to perform the calculation. Bars are not presented for groups sma=
ller
than 20 students for schools that did not conduct a probability sample.
If =
your
school conducted probability sampling, the first table below the graph (Tes=
t of
Statistical Significance) reports on whether the percentages of students
earning a High performance level were statistically different across groups.
Standard t-tests were conducted and the False Discovery Rate (Benjamini and Hochberg, 1995) procedure was used to c=
ontrol
the proportion of differences due to chance at 5%. Reading across the first=
row
of the first table, for example, entries of “>” indicate that
the group listed in the row likely has a larger percentage of students in t=
he
High performance level than the group indicated by the column heading. Entr=
ies
of “<” suggest a smaller percentage is likely. Entries of
“=3D” suggest that we cannot be confident that the percentages =
differ
for the groups. Values of “NA” correspond to bars removed from =
the
graph due to small numbers of students. The second table corresponds to the=
Middle
performance level, and the third table corresponds to the Low performance
level.
Add=
itional
reports on specified groups of interest follow, including subproficiencies
for each of the groups making up a comparison. Each comparison name and gro=
up
name is indicated in the report title. For example, a report might be called
“C1-1-1 School Year Performance Profile – F=
reshman”,
indicating that it contains subproficiency
information for the Freshman group within the School Year comparison. These
reports are interpreted for each group in the same way that Report A3 is
interpreted for all students (described above).
The=
series
of reports beginning with Report C2-1 presents each group’s overall I=
CT
performance as a mean. These are interpreted in the same fashion as similar
plots of score means for overall skill described in the text above.
About Define/Access
The DEFINE/ACCESS performance scale involves
behaviors related to formulating a research statement and using such a
statement to find and retrieve information from a variety of sources. Among
others, these sources might include web pages, databases, discussion groups,
email, or on-line descriptions of print media. Tasks contributing to this s=
cale
include those requiring examinees to ask questions in order to clarify an a=
mbiguous
assignment or need; formulate a tight, well-defined research question; thin=
k of
and combine search keywords to satisfy the requirements of a particular
research task; efficiently browse one or more resources to locate pertinent
information; and decide what types of resources might yield the most useful
information for a particular need.
Low
|
Middle
|
High
|
|
Generates or selects research questions or statements
regarding an information need that are: ·
Related to an information goal ·
Reasonably specific (e.g., event, issue, aspect, person, time peri=
od) ·
Largely essential to the information need ·
Rarely indicative of key concepts and terms |
Generates and recognizes research questions or stateme=
nts
of information needs that are: ·
Targeted at an information goal ·
Reasonably specific (e.g.,
event, issue, aspect, person, time period) ·
Largely essential to the information need ·
Somewhat indicative of key concepts and terms |
Generates and recognizes research questions or stateme=
nts
of information needs that are: ·
Targeted at an information goal ·
Appropriately specific (e.g.,&nbs=
p;
event, issue, aspect, person, time period) ·
Essential to the information need ·
Indicative of key concepts and terms |
|
Example: When faced with an
ambiguous course assignment, selects queries to submit to “professo=
r”
that are off-topic or fail |
Example: When faced with an
ambiguous course assignment, selects queries that are on-topic, |
Example: When faced with an
ambiguous course assignment, selects queries that elucidate the assignmen=
t |
|
When
presented with a limited pool of information resources, chooses a resource
that satisfies a clearly specified information need |
When
presented with a variety of information resources, sometimes chooses
resources that satisfy a given information need |
When
presented with a variety of information resources, usually chooses resour=
ces
that satisfy a given information need |
|
Uses search strategies for a given information need th=
at: ·
Employ some key search terms ·
Largely select accurate limits for searches ·
Employ ineffective syntax ·
Yield results that do not satisfy the need ·
Tend not to adapt searches to prior results, producing similar res=
ults
in subsequent searches |
Uses search strategies for a given information need th=
at: · =
Employ relevant key search t=
erms · =
Appropriately limit searches=
· =
Employ workable syntax that =
was
selected by the examinee · =
Yield more useful results th=
an not
useful results · =
Adapt to prior results, obta=
ining
different results on subsequent searches |
Uses search strategies for a given information need th=
at: ·
Target specific key search terms ·
Employ accurate selection of limits by examinee ·
Employ effective syntax that was generated or selected by the exam=
inee ·
Yield largely results that address the need ·
Adapt to prior results even for complicated search situations |
|
Example: When conducting a w=
eb or
database search, tends to use search terms and queries that yield a lot of
unrelated results |
Example: When conducting a w=
eb or
database search, tends to use search terms and queries that yield results
that satisfy the information need, but also include irrelevant results.
Adapts searches by adjusting search limiters (publication date or type) |
Example: When conducting a w=
eb or
database search, tends to use search queries that yield useful results
without a lot of irrelevant results and generates search terms that go be=
yond
the specific words of the information need. Adapts searches by including
terms discovered among previous search results |
|
When
browsing, appropriately selects links that follow an expressed clear path=
way
for the information need |
When
browsing, largely selects appropriate links for a pathway with more choic=
es
for the information need |
When
browsing, appropriately selects links for a pathway with more choices for=
the
information need |
About Manage/Integrate
The
MANAGE/INTEGRATE performance area involves the organization of information =
into
a provided scheme, as well as summarizing or synthesizing information. The
information might be email, files on a hard drive, results of a search, or
information contained in word-processed documents or spreadsheets. Tasks
contributing to this performance scale include asking examinees to identify=
how
a piece of information (email, file, etc.) fits into a category (e.g., email
subfolder or more complex hierarchy), and summarizing in a table or spreads=
heet
relative features of hotels, cars, or even prospective roommates. These tas=
ks
also include testing whether examinees demonstrate the foresight to prepare
information to make categorization, synthesis, or summarization more accura=
te
and efficient (e.g., sorting information before categorizing it).
Low
|
Middle
|
High
|
|
Usually
creates or selects an information classification scheme that allows effic=
ient
integration of information to meet a given need |
Creates
or selects an information classification scheme that allows efficient
integration of information to meet a given need |
Creates or selects an information classification scheme
that allows efficient integration of information to meet a given need |
|
Applies
classification schemes when categories are mutually exclusive |
Applies
classification schemes that involve categorization of materials into mult=
iple
locations |
Applies
classification schemes that involve categorization into multiple locations
and use of multiple tools |
|
Example: Accurately organizes multiple emails=
into
folders when each email can be unambiguously assigned to only one folder<=
/span> |
Example: Accurately organizes multiple emails=
into
folders even when some emails must be copied and placed in more than one
folder |
Example: When asked to organize multiple emai=
ls
into more than one folder, including folders on the hard drive, accurately
moves, copies, and/or deletes emails as specified |
|
Fails
to treat confidential information appropriately |
Treats
confidential information appropriately, but treats some non-confidential
information as if it were confidential |
Treats confidential or sensitive
information appropriately |
|
Summarizes or synthesizes information from multiple
sources into a single representation that: ·
Reflects information relevant to a clearly defined need ·
Rarely interprets the provided information correctly except in sim=
ple
circumstances ·
Shows accurate organization when a logical or=
der
is obvious |
Summarizes or synthesizes information from multiple
sources into a single representation that: ·
Reflects information to mostly satisfy the critical need ·
Sometimes interprets the provided information correctly, even when=
the
information need is not straightforward ·
Sometimes shows accurate organization, even when a logical order is
not obvious |
Summarizes or synthesizes information from multiple
sources into a single representation that: ·
Accurately and completely reflects relevant information even for a
vaguely defined need ·
Interprets the provided information correctly, even when the
information need is not straightforward · <=
/span>Shows accurate organization =
even
when a logical order is not obvious |
|
Sometimes
draws accurate conclusions from summary, but incorrectly justifies
conclusions |
Usually
draws accurate conclusions from summary, and sometimes accurately justifi=
es
conclusions |
Draws
complete and accurate conclusions from summary and accurately justifies
conclusions |
|
Example: When asked to choose
between two hotels for an event based on information on the hotels’=
web
sites, identifies the best hotel, but provides a rationale for the decisi=
on
that is inconsistent with the hotel features called for by the task |
Example: When asked to choose
between two hotels for an event based on the hotels’ web site,
identifies the best hotel, but provides a rationale that incorrectly refl=
ects
some hotel qualities called for in the task |
Example: When asked to choose
between two hotels for an event based on the hotels’ web site,
identifies the best hotel and provides a rationale that reflects all hotel
features called for in the task |
About Create/Communicate
The
CREATE/COMMUNICATE performance scale covers behaviors associated with
generating and adapting information to meet a specified need, including the
needs of a particular audience or a new delivery media (e.g., email,
presentation slide). Tasks that contribute to this performance scale include
generating electronic materials that express and support a main point with
accurately cited, meaningfully organized, and correctly formatted informati=
on.
Electronic materials include word-processed text, spreadsheets, graphs or
charts, web pages, or even presentation slides. Tasks also include those th=
at
ask examinees to reorganize or repurpose presentation slides for two distinct audiences, tailor a
graphical display made for one audience to be appropriate for a new audienc=
e,
and communicate appropriately with peers (less formal) and professors (more
formal). In all situations, whether creating materials or adapting them from
existing sources of information, examinees must show awareness of ethical a=
nd
legal issues by citing sources when necessary and using only public-domain
(i.e., not copyrighted) information.
Low
|
Middle
|
High
|
|
Successful
at creating only a limited range of electronic materials |
Creates
some types of electronic materials more effectively than others |
Effective
at creating a wide range of electronic materials |
|
Example: Accurately selects
material for a simple web page |
Example: Creates effective w=
ord
processed text, but has trouble creating effective presentation slides |
Example: Does well when |
|
In created materials, supports assertions with: ·
Some relevant information when the points to be made are
straightforward ·
Substantial extraneous information ·
No visual or organizational connections between elements in suppor=
ting
material and assertions ·
Incorrect citation of sources |
In created materials, supports assertions with: ·
Relevant information when the points to be made are straightforwar=
d ·
Some extraneous information ·
Some visual and organizational connections between elements in
supporting material and assertions ·
Most sources cited correctly |
In created materials, supports assertions with: ·
Relevant information even when multiple viewpoints are represented=
·
No extraneous information ·
Explicit visual and organizational connections between elements in
supporting material and assertions ·
Accurate citations of sources |
|
Example: Creates presentation slides with formatting and organization that
obscure the main points |
Example: Creates presentation
slides with formatting and organization that partially reveal |
Example: Creates presentation
slides whose main points are emphasized through formatting and organizati=
on |
|
Uses
the same materials regardless of audience needs, potentially employing an
inappropriate tone or a level of complexity inconsistent with the
audience’s needs |
Attempts
to customizes presentation of information to respond to an audience’=
;s
need but does so incompletely |
Effectively
customizes presentation of information to respond to an audience’s =
need
|
|
Example:
Presents graphs or |
Example: When adapting an ac=
ademic
abstract for a wider audience, might select text that retains some academ=
ic
jargon |
Example: When sending a stat=
us
email to teammates and a professor, avoids using overly informal language=
in
the |
About Evaluate
The
EVALUATE performance area covers behaviors associated with judging the
usefulness of information for a specific purpose, such as a particular rese=
arch
topic. In addition to critiquing the relevancy of information, tasks ask
examinees to discern the authority, timeliness, and bias of information
resources. Some tasks ask examinees to determine whether the totality of
information collected in a task (e.g., results of a literature search) is
sufficient for a need and, if not, to identify the gaps in the resources. <=
b>
Low
|
Middle
|
High
|
|
Sometimes
selects the most appropriate resources in terms of authority, point of vi=
ew,
and timeliness when these criteria are explicitly part of an information =
need |
Usually
selects the most appropriate resources in terms of authority, point of vi=
ew,
and timeliness when these criteria are explicitly part of an information
need |
Selects
the most appropriate resources in terms of authority, point of view, and
timeliness whether or not these criteria are explicitly stated |
|
Example: Selects just one of two
necessary web sites when required to find home pages with differing
viewpoints from a certain time period |
Example:
Given a classroom assignment, correctly judges the authority, objectivity,
and |
Example: Brows=
es a
set of web sites that review a play and, |
|
Sometimes
identifies relevant resources for an information need that has clearly
specified, simple criteria |
Identifies
relevant resources for an information need, and partially justifies the
selection |
Identifies
relevant resources for an information need, and accurately justifies the
selection |
|
Example:
When asked to justify selection of the best flyer for a particular need,
selects the best flyer and some accurate descriptions from a criteria |
Example:
When asked to browse for and justify selection of the best web site for a
particular need, selects a useful site and some accurate descriptions fro=
m a |
Example: When asked to justi=
fy a
ranking of three databases for a multiple-criteria need, identifies most =
of
the criteria met by each database
|
|
Often
recognizes whether resources are insufficient for an information need, but
fails to correctly identify what is lacking |
Recognizes
whether resources are insufficient for an information need, and is partia=
lly
accurate in identifying what is lacking |
Recognizes
whether resources are insufficient for an information need, and accurately
identifies what is lacking |
|
Example:
Given a description of a resource and the needs of a particular classroom
assignment, recognizes an insufficiency but |
Example:
Given a set of emails, recognizes what needed information |
Example: Given=
a set
of emails, recognizes which are complete |
|
Ignores
inflammatory language |
Identifies inflammatory language, sometimes censoring =
more
than is necessary |
Identifies
inflammatory language |