A qualitative analysis of publicly available Standards and Guidance about Digital Literacies in U.S. States

Before the COVID-19 pandemic, schools had been adopting digital instruction in many parts of the world. The concept of digital literacies has also been evolving in complexity alongside the digital technologies that support it. However, little is known about what guidance available to support various levels of government in supporting digital literacies alongside digital instruction in local schools. The purpose of this study was to determine what guidance for digital literacies U.S. state departments of education had made available through their websites to local schools just prior to the onset of the pandemic. Using qualitative content analysis techniques, digital literacies guidance information was located on U.S. state departments of education websites and analyzed. Most states did not indicate that they used guidance from professional organizations about digital literacies. The 16 states that did have guidance used standards from the International Society for Technology in Education (ISTE), which have not been positioned by the organization as digital literacies standards, but instead reflect traditional understandings of Information Computing Technology (ICT). Implications of this study highlight potential strategies educational ministries might use to acknowledge and support digital literacies.

Introduction
Digital technologies support meaning making and communication across modes (e.g., linguistic, visual, aural, and semiotic;Álvarez Valencia, 2016). These opportunities for making meaning with new technologies have been conceptualized as types of literacies because they rely on encoding/decoding processes to make meaning in social contexts (Hall, 1973(Hall, /1991. As technologies have developed to display increasingly complex texts with video, music, animation, and more, digital literacies now encompass a wide and ever-growing variety of strategies, knowledges, and skills. While there is no unanimous agreement about what outcomes for young people are preferable for acquiring digital literacies, proposed goals include: advancing social democracy, maintaining a workforce, and supporting ongoing innovation and quality of life (Coldwell-Neilson, 2017). Each of these goals is a high-stakes one, not just for the learners, but for societies and communities. Therefore, it is important for learners to have opportunities to use digital literacies in their schooling.
Explicit attention to digital literacies as part of digital learning has become more important to formal educational experiences due to the recent COVID-19 pandemic. As school buildings in the United States and across the globe closed, teachers in many countries turned to using digitized instructional materials as part of government directives to provide education (Natanson & Strauss, 2020). While the uses of instructional materials that required digital literacies were expanding even before COVID-19 closed school buildings, the pandemic drew attention to the need to understand efforts to promote digital literacies (Digital Learning Collaborative, 2019). Understanding the digital literacies policies that U.S. states had when the pandemic began can provide important information to the states themselves and to other governing entities when decision makers are ready to revise policies that anticipate long-term school disruptions (Rice & Zancanella, 2021).
This local policy making is critical in the U.S. because the U.S. does not have a national educational policy (U.S. Department of Education, 2021). Instead, individual states develop educational policies in accordance with federal guidance that is usually tied to federal monies. Some policies at the state level are developed in statelevel legislatures. Other policies are left to the discretion of state level educational officials hired to make such decisions. Therefore, it is to be expected that states would not be in alignment on any matter of educational policy. However, such differences provide the opportunity to see how different states have addressed important educational challenges so that states or countries with less developed policies can have models and examples in designing policies that work for them. Previous research suggests that many countries experience similar struggles to provide teachers with adequate policy guidance for using technologies generally and digital literacies specifically. For example, Dong and Newman (2018) found that teachers in China were using policy frameworks with inconsistent pedagogical recommendations for teaching digital literacies; Drajati et al., (2018) found that teachers in Indonesia had inconsistent access to common terms and language necessary for professional learning about technologies and information skills; and Luik et al. (2018) found that teachers in Estonia were integrating technologies into teaching without a pedagogical foundation. Considering findings from these studies, it seems useful to understand digital literacies policies and what organizations-with concomitant expertise and valuesmight be contributing to these.
To guide the leaders at the state level in the U.S., professional organizations with national and international reach have specialized in monitoring research and advocating for practice about digital literacies. Such organizations include the American Association of School Librarians (AASL), Computer Science Teachers of America (CSTA), International Society for Technology Education (ISTE), the National Council of Teachers of English (NCTE), and more. In addition, there are national consortiums that write standards and make recommendations about curriculum and instruction in the US, such as the National Governors Association Center for Best Practices and the Council of Chief State School Officers, which created the Common Core State Standards (2010). There was also a consortium of 26 states that formed the NextGen Science Standards (2013) with support from the National Science Teachers Association (NSTA), the American Association for the Advancement of Science (AAAS), and the National Research Council (NRC).
While it might seem to be a positive development that professional organizations and standards for digital literacies were abundant prior to the pandemic, there is no previous research about what guidance existed for digital literacies in different countries or in smaller government levels, such as a U.S. state. This gap extended to a lack of knowledge about how information, standards, and other materials from professional organizations might contribute to policy implementation in schools and classrooms. To address this gap, researchers investigated the following questions: 1. What professional organizations' guidance were U.S. states drawing on about digital literacies leading up to the pandemic? 2. What were the key descriptive features of that guidance?

Defining Digital Literacies
Various professional organizations may use various types of related terms to describe digital literacies. These terms suggest various purposes and values. Also, Digital literacies as a term, is ever-evolving due to its reliance on technologies that are ever expanding (Lankshear & Knobel, 2006). Due to this changeability, policies for digital literacies seem challenging to develop. In research projects, key terms for digital literacies also need to be expansive. Many of the literacies of the digital originated in the pre-digital period but have gained importance because of their links to more sophisticated practices (Bawden, 2008;Martin & Grudziecki, 2006). This has happened in much the same way that early household computers have a relationship with more modern machines. The more modern computers have greater memories and connective capacities and a larger array of options for using and producing text (Bawden, 2001(Bawden, , 2008. While the terminology around using and producing text with digital literacies has undergone considerable development, many of the same concepts that emerged in the 1960s governing technological knowledge and skills are still used today (Dakers, 2006;van Joolingen, 2004, slide 5). For example, machines like typewriters and even older computers still have screens and keyboards. Many of the same keystrokes are required to do the same task in the 1880s through to the 1980s and now in 2021.

Early definitions and terms
There were many attempts to define digital literacies that compare them to information literacies (Gilster, 1997;Eshet, 2002). Specifically, the Prague Declaration (UNESCO, 2003) emphasised the global importance of information literacy in the context of the "Information Society." The statement used words such as identify, locate, evaluate, organize, create, and use to convey its intentions. Specifically, the UNESCO document predicted a so-called new Information Society and indicated that Information Literacy was a basic human right.

Expansion of terms to Include Media Forms
Evaluating media has a critical bearing on individual health decisions, elections, and more (Potter, 2018). Media literacy studies have come to encompass understandings about how messages are constructed and interpreted, including understanding how characteristics of the author/sender and the receiver are crucial in understanding the meaning of the message and its content (Bawden, 2008;Martin & Grudziecki, 2006). Most recently, this includes social media and gaming (Rhodes & Robnolt, 2009). Part of understanding media requires understanding how images, text, and/or sound are used together to make meanings for various audiences (Blackall, 2005;Cameron, 2000;Mills, 2016). Multimodality also highlights the importance of symbolic systems and social semiotics (Kress & Leeuwen, 2006). Together, these elements of information, media, and mode shape what Lankshear & Knobel (2003) described as New Literacies and they include post typographic forms of texts, such as images, sounds, haptic manipulation, and more. These literacies help users access, interpret, criticise, and participate in culture and society (Kahn & Kellner, 2006;Kellner, 2002;Snyder, 2002;Tyner, 2014).

Recent definitions and expansions of Digital Literacies
More recently Çam & Kiyici (2017) merged elements from multiple time periods and frames, including ICT, information literacy, technological literacy, visual literacy, and media literacy. This definition summarised many of the repeating, overlapping ideas from previous definitions and provided the guidance for this study.
The term "digital literacy" today can be defined as the technical knowledge and skills necessary for the individuals who want to lead a productive life, to continue their personal development with lifelong learning activities, and to contribute positively to society. With this definition, the literacies incorporated in digital literacy can be listed as Information Literacy, Visual Literacy, Software Literacy, Technology Literacy, and Computer Literacy (p. 30).
For literacy researchers, there have been additional expansions on the idea of digital literacies to include more critical evaluation (Alvermann & Sanders, 2019;Digital Literacy High-Level Expert Group, 2008). In addition, Kellner (2002;2004) expanded the notion of literacies even further to include aspects of social power as a site for production and consumption of digital texts. This layer of criticality requires teachers to focus more on issues of audience and purpose than skills like typing or even coding as aspects of digital literacies. Using Kellner (2003) as a frame for critical digital literacies might be precarious in the current moment as many U.S. states are banning curriculum and instruction that even uses the word critical (Sawchuk, 2021). It is under these shifting definitions and expanding understandings about literacies that states are working to determine the importance of these and make educational policies about them. It is also in this context that digital literacies guidance at the state level, might use multiple terms and take on multiple forms.

Professional Organizations and Advocacy
Professional agencies in education in the U.S. and even globally in some cases have been developing various standards (e.g., ISTE, CTSA, Next Generation Science, NETEA, AASL) for the specific purpose of informing the research and practice. Moreover, organizations rely on entities to take up their standards to demonstrate that their organization is viable, deserving of their members, and worthy of additional members. Besides the practical matter of demonstrating their professional prowess, these professional organizations and the guidance they give should enable conversations about equity when they are taken up by entities such as states.
Professional organizations all over the world engage in developing standards or guidance to set expectations for professional practices as well as provide some managerial oversight (Wadmann et al., 2019). Within the U.S., and for education specifically, these organizations have advanced new standards, guided education policies, and influenced legislative and regulatory practices (Dauenhauer et al. 2019;2014;McGuinn, 2012).
Within such professional organizations, prominent members, boards, and/or task forces use research and perhaps federal guidance to craft standards and make recommendations for practice. Since the state leadership and the professional organizations might have the same access to members, research, and federal guidance, it stands to reason that most states would come to develop or adopt the similar standards from a narrow range of organizations, rather than each state crafting their own separate standards from multiple sources (DiMaggio & Powell, 1983). Moreover, states that are close together geographically may also be more likely to rely on the same professional organizations for standards. With these ideas about the relationship between professional organizations and leaders in-state entities, we undertook this policy scan study to explore what guidance for digital literacies U.S. state departments of education were providing to local schools.

Methodology
This policy scan drew on strategies from Ortiz et al. (2020) for obtaining information from state databases. The following policy scan activities were undertaken in 2018 through 2019: (1) open google search processes, (2) targeted searches on the state websites, (3) selection of information for inclusion, (4) content analysis of included items, and (5) verifying analysis for trustworthiness.

Open Google Searches
The first round of searching consisted of open Google searches. One researcher entered the terms "digital literacies [state name]", "technological literacies [state name]," "information technologies [state name]" and "information computing technologies [state name]" to allow for the widest search possible. From there, the links that were provided were followed to see if the search result was linked to the state office of education or was sponsored by them in any way. Whatever was linked was captured either as a link or downloaded as a document and stored with a file in the state's name.

Targeted state website searches
The second task was to go directly to the state website and search for guidance about digital/technological literacies. A second researcher engaged in this step. Where possible, searches were entered into search fields on the sites. Another strategy was to follow tabs and hyperlinks to literacy information and determine if there were any resources for digital/technological literacies. Resulting sites and documents were downloaded or captured as a link and then added to a file with the state's name. When this was done, the two researchers compared and compiled documents.

Selection of information for inclusion
Once the information from the folders had been compiled, it was time to determine which documents constituted guidance. The researchers used the following criteria to determine whether this was the case. First, the document or site needed to be formally prepared for public viewing. This usually indicated formatting, a title, and an explanation of where the document came from. Second, the document or site needed to specify that it was for use in planning instruction. Third, the document or site is needed to name specific student outcomes. These could be formal objectives, or they could be more general aims. Finally, the document needed to contain the phrase "digital literacies" or related terms to separate it from general technological implementation or assistance with technologically supported patterns like blended learning. A table was made with fields for each state. Items were added as they related to the research question. Specifically, researchers recorded content areas, the organizations within the state office of education or under their direction who were sponsoring the development of the document, and what professional organizations' information was cited in the document.

Content analysis of included items
The results underwent conventional content analysis to determine findings (Hsiu-Fang & Shannon, 2005). Content analysis leads to the "interpretation of content of text data through the systematic classification process of coding or identifying of themes or patterns" (Zhang & Wildemuth, 2009, p. 1278. Using a qualitative design for this content analysis emphasized "concepts rather than simply words" (Fraenkel & Wallen, 2011, p. 389) and also conveyed facts in a manner that was coherent and useful (Sandelowski, 2010). The content analysis for this project was conducted by having two researchers-one researcher from the initial identification and one entirely new researcher-move through the data in the first table to identify patterns. Several emerged. One was the prevalence of the ISTE standards as professional organizations whose work had been cited from the standards. The two researchers then made a new table of the ISTE states documenting whether the topic headings from ISTE were intact, the grade level articulations like those from ISTE had been used, the ISTE standards had been used as a primary framework or there were also other organizations cited, certain subjects or content areas were mentioned, and the definition of digital or technological literacy was included. The table was coded for patterns by the two researchers and then checked for accuracy by the researcher from the first round who participated in the identification of initial documents but did not participate in the initial content analysis.

Verifying analysis for trustworthiness
In qualitative research, trustworthiness reflects an effort to "control for potential biases that might be present throughout the design, implementation, and analysis of the study" (Bloomberg & Volpe, 2012, p176). The study was designed to allow other researchers to track "the processes and procedures used to collect and interpret the data" (p. 163). The account described above should enable other researchers to follow the same procedures and arrive at generally the same conclusions. During this process it was critical to ensure that major steps were taken and completed promptly so that websites would not change or that established issues of consensus among the researchers for inclusion would not be forgotten or shifted.
This process of working steadily also supported us in developing objectivity (Bloomberg & Volpe, 2018). The goal was to ensure that the findings come directly from the research rather than the biases and subjectivity of the researcher to the greatest extent possible. To ensure this, researchers used team processes where two researchers performed the identification task with a separate verification and then a team with one original researcher and a new researcher performed the analysis with a separate verification researcher. Further, one of the researchers was a digital literacies scholar and may have carried notions with them about what guidance should look like but was checked by another researcher who studied other aspects of technology in schools. A third researcher who did not research technology at all also contributed by providing an outsider view of what should constitute guidance about digital literacies (Hammersley, 2012). Thus, the findings presented should present a strong objective interpretation of the information that was discovered.

Findings
The purpose of this study was to learn what professional agencies U.S. states had been drawing on for guidance about digital literacies on their department of education websites. Each state department of education website was searched for guidance, but not every state had a plan. What follows is a description of the guidance that was found.

Overall description of findings
During the search process, the ISTE standards emerged as the most frequent source of professional guidance. Sixteen states used the ISTE standards as their primary source of guidance and support for digital literacies (See Fig. 1). Moreover, 11 states had adopted the standards as they were, with no modifications (Alabama, Alaska, Connecticut, Idaho, Mississippi, Nevada, Rhode Island, Vermont, Washington, West Virginia, Wisconsin). Nine states adopted the ISTE standards in concert with other professional organizations (Alabama, Minnesota, Mississippi, North Dakota, Oklahoma, Oregon, South Dakota, Washington, Wisconsin). These organizations will be discussed later. Finally, eight states even included the same grade level articulations as those that ISTE uses (Alabama, Alaska, Idaho, Nevada, South Dakota, Washington, West Virginia, Wisconsin).
The states that used the ISTE standards placed them in context with a variety of subject matter orientations. Some states used the standards in more than one subject matter orientation: computer science (Alabama, Washington, West Virginia, Wisconsin); library science (Connecticut, Oregon, Wisconsin); general technology and information science (Idaho, Nevada, Washington, Wisconsin); Other sciences (Wash- Fig. 1 States that use ISTE standards for digital literacies ington, Wisconsin); English language arts/general literacy (Mississippi, Washington, Wisconsin); mathematics (Mississippi, Washington, Wisconsin); and social science (Washington).
A few states that featured professional guidance about digital literacies used standards from groups other than ISTE. For example, the state of Washington had a document that incorporated guidance from NSCC, CTSA, and Next Generation Science Standards. Also, Washington referenced the largest number of subject matter specialties, including science, math, and technology, as well as library sciences, social studies, and English language arts. Table 1 contains sample descriptions from various states.

Exploration of several States' plans
States that incorporated ISTE standards to address digital literacies generally agreed on the importance of preparing students for digital citizenship as a critical aspect of digital literacies. However, states took different paths to the creation of their digital literacy guidance. Two examples come from Alabama, a politically conservative state in the southeastern U.S. and Connecticut, a politically progressive state in the northeastern part of the U.S. Alabama spends less money per pupil than most states and has a lower graduation rate than many other states. Connecticut spends more money per pupil than most states and has a higher graduation rate and students score higher on college entrance tests than in most other states. These differences provide an opportunity for comparison and contrast of their plans.

Alabama
The Alabama public school system (prekindergarten through grade 12) operates within smaller districts governed by locally elected school boards and superintendents (Ballotpedia, 2021a). The state has about 750,000 students enrolled in roughly 1,700 schools in 173 school districts (Ballotpedia, 2021a). The state has about 52,000 teachers in their schools (Ballotpedia, 2021a). On average Alabama spends close to $9,000 per pupil and ranked it 39th highest in the nation in spending (Ballotpedia, 2021a). The state's graduation rate hovers around 80%.
In terms of digital access, about 88.6% of all Alabamians have access to highspeed internet, defined as internet with download speeds of 25 megabits per second or faster (Smith, 2020). There are about 475,000 people in Alabama without access to high-speed internet, including large swathes of rural Alabama (Smith, 2020). In fact, nine of Alabama's 67 counties have less than 30% access to broadband, all of which are counties where many Black families live, some of which have poverty rates of greater than 30% (Smith, 2020). In addition, the US census bureau estimates only 73.3% of households subscribed to broadband internet between 2014 and 2018, suggesting that even when households have access to broadband, they may not be able to afford it (Smith, 2020). The governor of Alabama allocated 100 million dollars of federal disaster relief money to broadband access and schools partnered with local business to provide free Wi-Fi to schools for remote instruction. Given these circumstances, many K-12 students in the state of Alabama need access to the internet

North Dakota
The North Dakota State content standards offer instructional guidance in core curriculum areas, while at the same time, they allow for, indeed encourage, a dynamic and living curriculum created at the local school district level. To ensure educational relevance, our state's content standards are (1) based on academic standards developed nationally by various professional education associations, (2) validated by our state's best educators based on classroom experience and local community expectations, and (3) widely supported by state and national education policymakers. Oklahoma The Oklahoma Society for Technology in Education (OKSTE) provides instructional support, innovative leadership, and professional development as learning communities fuse the science of teaching and learning with EdTech. OKSTE is an ISTE affiliate and will use that relationship to create cadres of learning communities that are innovation leaders in the authentic integration of EdTech in Oklahoma classrooms.

Rhode Island
The International Society of Technology in Education (ISTE) Standards act as a roadmap for educators and education leaders looking to re-engineer their schools and classrooms for digital age learning. Vermont The Vermont State Board of Education adopted the International Standards for Technology Education (ISTE) Standards for student learning. The adoption of this framework is aligned with the state's Education Quality Standards. These standards outline what Vermont students should know and be able to do with respect to information technology and will guide and inform the work of our schools as they prepare students for college and careers that have been dramatically transformed by information technology. Washington Defines digital and media literacies and makes recommendations for legislative actions that support policies and practices.

West Virginia
The West Virginia Board of Education and the West Virginia Department of Education are pleased to present Policy 2520.14, 21st Century Learning Skills and Technology Tools Content Standards and Objectives for West Virginia Schools. The West Virginia Standards for 21st Century Learning includes 21st century content standards and objectives as well as 21st century standards and objectives for learning skills and technology tools. This broadened scope of curriculum is built on the firm belief that quality engaging instruction must be built on a curriculum that triangulates rigorous 21 st century content, 21 st century learning skills and the use of 21st century technology tools.
alongside access to digital literacies. Also, K-12 students in the state seem to need stronger educational support overall. Alabama created a task force that was charged with conceptualizing and creating a Course of study for digital literacies and computer science (Richardson, n.d). The task force in Alabama that created their pre-pandemic digital literacies policy document consisted of stakeholders that represented multiple areas of the state of Alabama's educational system. The task force included members from local Boards of Education, faculty members from local universities, school technology specialists, and one representative from industry. The industries represented were technological in nature, such as a company that sold email sorting software.
The Alabama document focused drew on the ISTE standards, saying: The International Society for Technology in Education (ISTE) Standards for Students emphasize the skills and qualities we want to foster in students, enabling them to engage and thrive in a connected, digital world (Richardson, n.d., p.1).
This does not say anything explicit about literacy, but rather that the skills and the ethos of the standards are the reason for their inclusion in the development of the course of study. For defining digital literacies, the group members generated the following definition: Digitally literate students can use technology responsibly and appropriately to create, collaborate, think critically, and apply algorithmic processes. They can access and evaluate information to gain lifelong knowledge and skills in all subject areas. Alabama includes computational literacy in its definition of digital literacy. Our goal is digital and computational literacy for all Alabama students (Richardson,n.d.,p. 1).
Their definition includes many of the goals advanced for digital literacies, including advancing social democracy through the goal to "collaborate" and "use technology responsibly" (Mora, 2016), maintaining a workforce through the emphasis on "computational literacy" (Pothier & Condon, 2019), and supporting ongoing innovation and quality of life through "lifelong knowledge and skills" (Coldwell-Nelson, 2017). In this course of study, the team who created the document hybridized many ideas about digital literacies and digital learning to create a guidance document for their state.

Connecticut
The Connecticut public school system (prekindergarten through grade 12) was operating within districts governed by locally elected school boards and superintendents (Ballotpedia, 2021b). Connecticut has about 560,000 students enrolled in a total of 1,200 schools in 200 school districts (Ballotpedia, 2021b). There were about 44,000 teachers in the public schools (Ballotpedia, 2021b). On average Connecticut spends about $17,000 per pupil per year, which ranked it fifth highest in the nation in spending (Ballotpedia, 2021b). The state's graduation rate is about 86% (Ballotpedia, 2021b).
During the pandemic, Connecticut spent 60 million dollars in federal monies and private donations to ensure that all students had an internet-ready device for school. Prior to the pandemic, about half of Connecticut's student population lacked an internet connection and/or an internet-ready device in their home (Thomas & Wat-son, 2020). Many of these families without access were people of color, particularly Black and Hispanic/Latinx families (Thomas & Watson, 2020). Even so, thousands of school children were left without a broadband connection when remote schooling resumed in the fall (Thomas & Watson, 2020). These facts illustrate that even in states that spent more money on education and generally had better outcomes, there were large disparities when it comes to access to the internet, devices, and thus, digital literacies.
In Connecticut the department of education created a document called Charting New Frontiers in Student-Centered Learning (Casey et al., 2017). To create the document, there were multiple councils assembled into one large advisory committee. These councils included topics of data and privacy, digital learning, and infrastructure. Individuals who participated in these councils were representatives from public schools-mostly at the administrative level, higher education, library directors, law firms, data security officers from universities, and technology vendors. The town manager of Coventry, Connecticut also participated as a community member. Overall, there was far less participation from school representatives, but more from other groups.
In terms of digital literacies, the plan offered no direct definition, but the document expressed the intention for libraries to be sites of digital literacies: Connecticut's libraries offer digital and media literacy, cybersecurity, and other technology training to students of all ages, helping to bring about a more literate citizenry that can better understand and support the power of technology to deepen and personalize learning (Casey et al., 2017, p. 47).
The document includes references to media, data privacy, as well as ICT skills through "training" to students. The goals for democratic participation are also evident (Coldwell-Nelson, 2017). What is different about this document from Alabama, other than the greater emphasis on digital skills over digital literacies. In Alabama, the definition offered had a more community-centered approach and a stronger articulation of lifelong benefits to the students. Also, the Connecticut document does not mention ISTE at all, but instead includes several state-based technology commissions and the national organization Innovation Partners of America.
Together, these two examples demonstrated how states were offering very different documents-some with very narrow definitions in terms of content, and some with more open definitions going into the pandemic. Importantly, these extant policies were the baseline or starting point for what states had been providing to schools before massive remote learning set in. The comparison also highlights the various ways that digital literacies might be included in guidance documents to varying degrees while still expressing similar values for students to understand technologies and engage with them in ways that benefit them and possibility their communities.

Discussion
The purpose of this research was to investigate the integration of professional guidance about digital literacies into U.S. state education policies. Researchers wanted to know whose guidance states were relying on and how the states had incorporated that guidance within the subject matter and grade levels. Limitations to this study included the fact that states may post, update, move, or remove information at their will so not all information located during the search period may be located now, but available data can be provided to others upon request. With keyword searches, there is always a chance that additional keywording would have yielded additional information. Our purpose of learning about guidance for digital literacies focused on connecting information to literacy pages on state department of education websites, but it is possible that state technology education officials have taken up digital literacies without posting information online as evidence. Also, state officials change periodically or shift responsibilities, making it difficult to identify persons who might have information or who may be posting it.
While it is not a limitation, but rather an issue of scope, we remind readers that we did not gather data about how those policies might have changed during the pandemic. We might anticipate that not very much policy making directly about digital literacy occurred, however, since it would seem more likely that states were scrambling just to get remote online learning set up so that some instruction could be given. What is clear from an analysis of the data is that states took a variety of approaches that in future work can be examined by policy makers to see if there is need of revision.
Finally, it is important to remember that ISTE standards are used by many states, but that the purpose of this research was to find guidance for digital literacies. Not every state that used ISTE standards placed the standards alongside notions of digital literaciesand not every state that used ISTE standards as digital literacies has made strong connections between the standards and digital literacies.

What Professional Organisations' Guidance do U.S. States draw on about Digital Literacies?
Although 16 states used the ISTE standards somewhere in their digital literacy guidance and a handful used standards from other places, over half of the states did not mention any organization's guidance in connection with digital literacies. This could mean that states were not prioritizing digital literacies or any of its related concepts (information literacies, technological literacies). It could also mean that states did not know where to go for guidance. Afterall, Lankshear and Knobel (2006) pointed out that defining and understanding digital literacies outside of academic spaces is an imprecise endeavor. Indeed, in the two case study states, many families of color and families in rural areas were without internet access and devices, which might take priority for digital literacies away from officials in favor of other needs, including basic needs for connections and devices. However, when states encountered difficulties moving to digital learning, some of those challenges might come from an apparent lack of clarity about what are digital literacies and what constitutes digital knowledge and skills (Cameron, 2000;Mills, 2016). Reciprocally, one needs digital literacies to advocate for and leverage access to the internet and devices, but one also needs an internet connection and a device to use and develop digital literacies.
For states that had guidance, many chose the ISTE standards. This was an interesting choice because the ISTE standards do not explicitly claim to be literacy stan-dards. In an explanation of the recent revision of the standards, Trust (2018), who was one a member of the revision team, noted that the standards were focused on helping teachers in using technology to "support student learning and creative thinking, design digital age activities and assessments, model digital work, promote and model digital citizenship, and engage in professional growth and leadership" (p. 1). Notice the similarity of these standards to many of van Joolingen's (2004) definition of ICT literacies as: the interest, attitude, and ability of individuals to appropriately use digital technology and communication tools to access, manage, integrate and evaluate information, construct new knowledge, and communicate with others to participate effectively in society (slide 5).
Reading the information from Trust (2018) and van Joolingen (2004) together, it is evident that ISTE standards touch on many of the same priorities as ICT literacies although ISTE has never claimed to be a literacy organization or claimed the ISTE standards are about literacies. Such circumstances again, raise the issue as to whether there is a meaningful distinction at this point in technological development between digital literacies and the knowledge, skills, and dispositions to engage in digital learning generally (Cameron, 2000;Mills, 2016). There may not be a meaningful distinction for state-level officials who design, approve, and promote this guidance.
Other countries striving to support digital learning might also overlook digital literacies as a distinct need or embed it within general learning without reflection.
Further, while there was a wide variety of guidance taken up by states who were using guidance, there were some notable organizations that were not represented. One such example was the National Council of Teachers of English, which has a committee called Digital Literacies in Teacher Education (D-LITE). The D-LITE group has published several iterations of Beliefs about Integrating Technology into the English Language Arts Classroom (Lynch et al., 2018). Other countries might also have organizations that have guidance that has not been accessed by government leaders and could prove helpful.
Another challenge in the U.S. and abroad might be the participation of educational technology vendors in policy making without proper transparency. Such participation was more of an issue in Connecticut where there are many headquarters for educational technology companies versus Alabama, which prioritized wide participation from local school experts. Leaders in the U.S. and other countries might consider who is participating in these policy discussions, what their motives might be, and what can be done, not necessarily to discourage or eliminate vendor participation, but to declare it and understand its limitations the same as any other group. For example, many other countries struggle with non-governmental organizations (NGOs) that sometimes come into countries and promise support for underserved groups; however, people from those countries do not always trust that they are not just seeking government money from poor countries as a source of profit (Author, 2019).
In summary, the fact that the states that do have plans mostly drew their guidance from the same organization, even though that was not explicitly a literacy organization, supports the theory that many U.S. state leaders have converged on similar policies to ease the burden of trying to generate totally separate guidance within their states (DiMaggio & Powell, 1983). While there was only one state that had execu-tive representation from ISTE on their board (Nevada), many of the planning board members may be members of ISTE and their familiarity with ISTE many have led them to think that adopting the standards was time and money saving. Again, the researchers of this study find nothing wrong with the ISTE standards-they merely question whether their wide use as literacy standards occurred because there was no real distinction between digital literacies and digital learning skills in policy makers' minds, or these leaders were unable to reflect on the differences due to time, cost, or other pressures. This is also an important question in the U.S. and internationally when research studies use terms like ICT and e-learning as catch-all phrases for any learning that uses digital elements. As research, policy, and practice communities, we should be asking what the value is in having and using certain terms with precision, or if the terms should be left to local decision makers.

What are the key descriptive features of the Guidance documents?
As for question 2 about how professional guidance has been integrated, the answer is that in cases where the standards are used, they are used in their entirety. Moreover, the grade level distinctions recommended by ISTE were often included. Only a few states, such as Minnesota, Washington, and Alabama, had developed comprehensive digital literacies guidance that drew on multiple sources and had been customized to the needs of the states. Regardless of the unique needs that probably exist in every state, the common needs are the ones that are in the best position to be met because they enable complete adoption of standards that already exist and again, carry the perception of saved time and money (DiMaggio & Powell, 1983). While it is beyond the scope of this research to say how and whether local schools used the guidance, it is interesting to consider how participation in the creation of the plans might shape later use of them in the places that matter: classrooms. This also seems important in the international arena where there may be differing levels of federal oversight of education.

Implications
Now that the pandemic has changed education and offered insight into the best and worst of digital education experiences for school children and their teachers, there is a need to make stronger plans for digital learning during times of disruption and beyond (Rice & Zancanella 2021). Along these lines, the current study has several important implications for practice, research, and policy.

Practice
High quality digital literacy learning is not an accident and does not seem to be coercible in the moment of crisis (Rice & Zancanella, 2021). Therefore, educational agencies need to study guidance from professional organizations in their respective countries about digital literacies use to craft sound policies and practices. We also feel that these conversations can include all digital learning, but that some explicit conversation about digital literacies is important, given the interrelationship between digital literacies and digital learning. It is our contention that digital literacies is more than just ICT knowledge repackaged from the 1960 and 1970s (Martin & Grudziecki, 2006). In a true frame of digital literacies, practices that are oriented around analysis and evaluation of digital texts and tools, alongside awareness of audience and purpose for what is composed in digital spaces are most important (Alvermann & Sanders, 2019;Kellner, 2002;2004;Mills, 2016).
Finally, U.S. state agencies and policy making agencies in other countries should consider how to make this guidance appropriate for the schools in their stewardship (U.S. Department of Education, 2021). There seems to be strong opportunities to make plans that draw on local strengths and expertise, such as was done in Alabama, since there has been so little federal guidance about digital literacies in the U.S. and so many national organizations with advice seem to stand at the ready. For countries without a wealth of professional organizations, leaders might consider looking to a geographic neighbor with shared history and values as a starting point. For example, Canada developed a nationwide overview of the digital literacy strengths and needs across their provinces and outlined steps that might be taken to develop policies at the level of provinces. (Hoechsmann & DeWaard 2015). Their work might be useful for countries like the U.S. that share some aspects of political organization, history, and geographic proximity. In turn, Canadian officials might consider what professional organizations in their country or internationally might have additional insight for policies that leverage the strengths for their various provinces.

Research
Future research projects should look at how states with various types of guidance from various agencies fared during the transition to emergency remote learning during the COVID-19 pandemic (Education Week, 2020). This research might be done in countries outside the U.S. with whatever guidance these political units were using. Besides achievement data, perception work might be useful in finding out whether educators within the state at various levels know about the guidance or use it. Moreover, some individual school districts and agencies might have their plans that draw on various sources of guidance. It seems worth investigating how these plans are formed and how professional organizations' guidance makes its way (or does not) into the plans. In other countries, similar work might determine what educators in classrooms know about the policies for digital literacies (if there be any) in their jurisdictions. It might also be helpful to look at what aspects of guidance are most important to set a foundation for digital literacies development and digital learning in schools. For example, findings from a study in Ireland suggest that broadband access might be a critical foundational step for establishing engagement for digital learning generally (Mac Domhnaill et al., 2021).
Future research about professional learning about digital literacies provided by U.S. states or other agencies and local school entities internationally might shed additional light on how professional guidance makes it to teachers and into their practice. For example, it seems important to understand how teachers understand guidance about digital and technology education in cases where digital literacies and its accompanying ideologies about power construction and community practices are included and when they are not (Kellner, 2002;2004). This is especially important considering that many U.S. states and many countries were scrambling to acquire internet connections and devices for students when the COVID-19 pandemic shut down many school buildings (Mac Domhnaill et al., 2021;Smith, 2020;Thomas & Watson, 2020). Who receives devices and access and why they have it are part of the critical digital literacies that students need (Kellner, 2002;2004).

Policy
While ISTE redesigned its Standards for Educators regarding how to use technology to learn, collaborate, lead, and empower students based on input solicited from more than 2,000 educators and administrators, generally, the relationship between standards and academic research remains unclear (Trust, 2018). While practice to policy to research is certainly a possible pathway to understanding, there are alternative pathways that begin with collaborative research with practitioners and then move to policy before recommending procedures for practice.
While there were U.S. states that used ISTE standards that did not expressly connect the standards with digital literacies, several of them did. In terms of policies, ISTE might consider how many state-level agencies have adopted their standards into their digital literacies guidance. What should ISTE do with this information? They might be more explicit about how their standards promote digital literacies. They might also distance themselves from literacies and focus more on their work as a general technological organization. Other political units and entities might also consider reviewing the guidance they give to ensure that digital literacies are specifically addressed according to how it is conceptualized and applied in local settings. The questions remain: Is there a meaningful distinction between digital literacies and digital learning in the minds of practitioners and policy makers? Should there be?

Conclusion
The purpose of this study was to understand what professional agencies were providing the guidance that U.S. states were adding to their state department of education websites as part of their support for local schools. By locating and analyzing the various plans, the ISTE standards for technology learning were the most popular, but most states offer no guidance at all. However, the states that have adopted the ISTE standards, did seem to follow the isomorphic pattern of geographical clustering (Fig. 1). These findings suggest that leaders in various regions should consider including guidance that is focused on literacies directly, even when they have guidance about learning generally (Alvermann & Sanders, 2019;Kellner, 2002;2004). With the COVID-19 pandemic's disruption of schooling and the continuation of the trend toward the digital instructional materials, now is the time for states to make strategic decisions about how to include digital literacies in their formal guidance.
Data Availability Some data may be available upon request.


Toward Reliability in the NISQ Era: Robust Interval Guarantee for Quantum Measurements on Approximate States

Quantum computation in noisy, near-term implementations holds promise across multiple domains ranging from chemistry and many-body physics to machine learning, optimization, and finance. However, experimental and algorithmic shortcomings such as noise and decoherence lead to the preparation of imperfect states which differ from the ideal state and hence lead to faulty measurement outcomes, ultimately undermining the reliability of near-term quantum algorithms. It is thus crucial to accurately quantify and bound these errors via intervals which are guaranteed to contain the ideal measurement outcomes. To address this desire, based on the formulation as a semidefinite program, we derive such robustness intervals for the expectation values of quantum observables using only their first moment and the fidelity to the ideal state. In addition, to get tighter bounds, we take into account second moments and extend bounds for pure states based on the non-negativity of Gram matrices to mixed states, thus enabling their applicability in the NISQ era where noisy scenarios are prevalent. Finally, we demonstrate our results in the context of the variational quantum eigensolver (VQE) on noisy and noiseless simulations.

Quantum computation in noisy, near-term implementations holds promise across multiple domains ranging from chemistry and many-body physics to machine learning, optimization, and finance. However, experimental and algorithmic shortcomings such as noise and decoherence lead to the preparation of imperfect states which differ from the ideal state and hence lead to faulty measurement outcomes, ultimately undermining the reliability of near-term quantum algorithms. It is thus crucial to accurately quantify and bound these errors via intervals which are guaranteed to contain the ideal measurement outcomes. To address this desire, based on the formulation as a semidefinite program, we derive such robustness intervals for the expectation values of quantum observables using only their first moment and the fidelity to the ideal state. In addition, to get tighter bounds, we take into account second moments and extend bounds for pure states based on the non-negativity of Gram matrices to mixed states, thus enabling their applicability in the NISQ era where noisy scenarios are prevalent. Finally, we demonstrate our results in the context of the variational quantum eigensolver (VQE) on noisy and noiseless simulations.

I. INTRODUCTION
Today's quantum computers are characterized by a low count of noisy qubits performing imperfect operations in a limited coherence time. In this era of quantum computing, the noisy intermediate-scale quantum (NISQ) era [1], researchers and practitioners alike strive to heuristically harness limited quantum resources in order to solve classically difficult problems and also to benchmark and potentially develop new quantum subroutines. A typical pattern of these NISQ algorithms [2], exemplified by the seminal variational quantum eigensolver (VQE) [3] and quantum approximate optimization algorithm (QAOA) [4], consists of the preparation of ansatz states with a parameterized unitary circuit followed by useful classical output being extracted by means of quantum measurements, more generally as expectation value of quantum observables through repeated measurements.
The promising potential of these NISQ algorithms spans across a wide spectrum of applications, ranging from quantum chemistry, many-body physics, and machine learning to optimization and finance [2]. However, as a consequence of their heuristic nature and the prevalent imperfections in near-term implementation, it is difficult to guarantee the degree of validity of the output. This unfortunate hindrance practically arises from various sources such as circuit noise and decoherence [1], lim- * aspuru@utoronto.ca † ce.zhang@inf.ethz.ch ‡ zhikuan.zhao@inf.ethz.ch ited expressibility of ansatze [5,6], barren plateaus during optimization in variational hybrid quantum-classical algorithms [7][8][9], measurement noise and other experimental imperfections [10,11]. To determine the usefulness of a given NISQ application, it is thus crucial to quantify the error on the final output in the presence of a multitude of the aforementioned sources of imperfection.
In this work, we endeavour to systematically certify the reliability of quantum algorithms by deriving robustness bounds for estimates of expectation values arising from imperfect approximations of a target state. To that end, based on analytical solutions to a semidefinite program (SDP), we present lower and upper bounds to quantum expectation values of observables which depend only on the fidelity with the target state and postprocessing of previously obtained measurement results. In addition, to obtain tighter bounds, we take into account higher moments of the observable by leveraging the Gramian method for pure states [12] and extend it to derive bounds which are valid for general density operators. This extension from pure to mixed states is particularly relevant in the NISQ era as it allows to derive robustness bounds which are valid in noisy scenarios. We then apply these bounds numerically and run noisy and noiseless simulations of VQE for finding ground state energies of electronic structure Hamiltonians of several molecules. Finally, we implement our results in the open source Tequila [13] library.
The remainder of the paper is organised as follows. In Sec. II, we present our main results, namely, the bounds based on SDP and the Gramian method. In Sec. III, we present our numerical simulations and explain the applicability of our bounds in the context of VQE. Sec. III B highlights the implementation in Tequila and concluding remarks are given in Sec. IV.

II. ROBUSTNESS INTERVALS
In this section, we present our main results, namely robustness intervals for quantum expectation values of observables. We first introduce the notation, summarize the technical results, and then proceed with presenting the robustness intervals.

A. Notation
The Hilbert space corresponding to the quantum system of interest is denoted by H ≡ C d with dimension d = 2 n . We use the Dirac notation for quantum states, i.e. elements of H are written as kets ψ⟩ ∈ H with the dual written as a bra ⟨ψ . The space of linear operators acting on elements of H is denoted by L(H) and elements thereof are written in capital letters A ∈ L(H). The set of density operators on H is written as S(H) ⊂ L(H) and lower case greek letters are used to denote its elements σ ∈ S(H) which are positive semidefinite and have trace equal to 1. For an element A ∈ L(H) we write A ≥ 0 if it is positive semidefinite, A T stands for its transpose, and A † is the adjoint. Expectation values of observables, i.e. Hermitian operators A ∈ L(H), are written as ⟨A⟩ σ = Tr[Aσ] for some σ ∈ S(H). The variance of an observable is given by (∆A σ ) 2 = ⟨A 2 ⟩ σ − ⟨A⟩ 2 σ . We write for the trace norm of an operator A ∈ L(H). The fidelity between quantum states σ, ρ ∈ S(H) is defined as F(ρ, σ) = max ψρ, ψσ ⟨ψ ρ ψ σ ⟩ 2 where the maximum is taken over all purifications of ρ and σ. For pure states, the fidelity reduces to the squared overlap F( ψ⟩⟨ψ , φ⟩⟨φ ) = ⟨ψ φ⟩ 2 . Finally, the real part of a complex number z ∈ C is written as R(z) and the imaginary part as I(z).

B. Summary of technical results
In this work, we present lower and upper bounds on the true expectation value ⟨A⟩ σ of an observable A under a target state σ, in terms of an approximate state ρ. To that end, we employ two different techniques, each with its advantages and disadvantages in terms of efficiency and accuracy. The first technique is based on the formulation as an SDP and makes use of a closed form solution of optimal type-II error probabilities from quantum hypothesis testing [17]. The second technique is based on the non-negativity of the determinant of Gramian matrices for a suitable collection of vectors. This second technique was initially proposed by Weinhold [12] in the context of pure states. Using Uhlmann's Theorem [18], Bond dissociation curves and robustness interval (RI) for Lithium Hydride in a basis-set-free approach [14,15]. The exact, theoretical energies are shown in black, the energy estimates provided by a noisy VQE with an UpCCGSD Ansatz [16] is shown in blue. The robustness interval is guaranteed to contain the true ground state energy and is based on the Gramian eigenvalue bounds for mixed states (Theorem 3).
which relates the fidelity between two mixed states to the trace norm, we extend these results to mixed states. This ultimately justifies their applicability in the current NISQ era, where the assumption of a closed quantum system is violated and one needs to make use of the density operator formalism to accurately model these states and their evolutions.
In Table I, we summarize all the results, together with the conditions under which they apply and the quantities that are covered. Fig. 1 shows the ground state energies of molecular Lithium Hydride in the basis-set-free approach of Refs. [14,15] with energy estimates provided by VQE with an UpCCGSD ansatz. The lower and upper bounds on the true energy are obtained via the Gramian method from Theorem 3.

C. Bounds via semidefinite programming
We present a robustness interval which is based on a formulation of the problem as a semidefinite program. This interval only requires the knowledge of the first moment of the operator A and a lower bound to the fidelity F(ρ, σ) between the target state σ and the approximation ρ. The idea is to define the problem as a semidefinite program (SDP) and then, using results from quantum hypothesis testing, derive a corresponding closed form solution. This yields upper and lower bounds on the true expectation ⟨A⟩ σ which can be expressed in terms of the fidelity F(ρ, σ) and the expectation ⟨A⟩ ρ under the approximation ρ. We remark that, although used in a different context, this technique is conceptually similar to the result presented in Theorem 1 of Ref. [17] for ad-
versarial robustness of quantum classification. Formally, we present the following theorem: , the lower bound of ⟨A⟩ σ can be expressed as Similarly, for ≤ 1 2 (1 − ⟨A⟩ ρ ), the upper bound of ⟨A⟩ σ becomes In a nutshell, this result is obtained by first formulating the problem as an SDP in which the quantity ⟨Λ⟩ σ is maximized (or minimized in case of the lower bound) over all operators Λ with −1 ≤ Λ ≤ 1 and whose expectation value is equal to ⟨Λ⟩ ρ = ⟨A⟩ ρ . Formally, the upper bound can be expressed as where for the lower bound the maximization is replaced with a minimization. A closed form solution to these SDPs can then be derived by expressing it in terms of a binary quantum hypothesis testing problem. Thus, Theorem 1 provides an interval which is guaranteed to contain the true expectation value ⟨A⟩ σ of an observable A ∈ L(H) under a target state σ. In other words, it provides thresholds on the value of that expectation value when only an approximation of the state σ, the state ρ, is provided. Stemming from the formulation as an SDP, these bounds are tight for pure states in the sense that, for each bound, there exists an observable A with expectation ⟨A⟩ ρ and whose expectation under σ saturates the bound. In Appendix A 2, we give a formal, constructive proof for this statement. Furthermore, in practice, it is typically not feasible to measure the exact expectation value ⟨A⟩ ρ due to finite sampling errors, measurement noise, and other experimental imperfections. For this reason, one needs to rely on bounds which contain the exact value with high probability. This can be accounted for in the bounds from Theorem 1 by noting that they are monotonic in ⟨A⟩ ρ , whose exact value can thus be replaced by a corresponding lower or upper bound. Finally, it is worth noting that, if one has access to an estimate of the fidelity, i.e. some > 0 with F(ρ, σ) ≥ 1 − , this interval can be calculated by merely postprocessing of previous measurement results, and hence does not cause any computational overhead.

D. Bounds via non-negativity of the Gramian
Here we employ a different technique to derive robustness bounds, taking into account the variance of the observable as an additional piece of information. The method is based on the non-negativity of the Gramian and was pioneered by Weinhold [12]. We first give a brief overview of the Gramian method and then present the extension to mixed states. This extension is important as the restriction to pure states hinders the applicability of this method in practice and, in particular, in the current NISQ era, where one often has to deal with noisy states that are expressed as probabilistic ensembles of pure states in the density operator formalism.

Review of the Gramian method
Consider a Hermitian operator A ∈ L(H), a target state ψ⟩ and an approximation of this state φ⟩. The Gram matrix for the vectors ψ⟩, φ⟩, A φ⟩ is given by where, without loss of generality, it is assumed that the ovelap ⟨ψ φ⟩ is real (otherwise multiply each state by a global phase). Since Gram matrices are positive semidefinite (e.g., Theorem 7.2.10 in [19]), their determinants are non-negative, det(G) ≥ 0. The function det(G) is a quadratic polynomial in R(⟨ψ A φ⟩) where (∆A φ ) 2 = ⟨φ A 2 φ⟩ − ⟨φ A φ⟩ 2 is the variance of A under φ⟩. This polynomial vanishes when R(⟨ψ A φ⟩) takes the values and the non-negativity of G thus limits the permissible values of R(⟨ψ A φ⟩) to be within these boundaries.
Since I(⟨ψ A φ⟩) 2 ≥ 0 we have the bounds Starting from these inequalities and whithin this Gramian approach, a bounds for expectation values of quantum observables have been derived for pure states [12,[20][21][22][23][24][25][26] and in the context of classical methods for quantum chemistry. While certainly useful, this leaves a gap for these bounds to be applied in the NISQ era where noise is prevalent and quantum states and their evolutions are described by density operators. In the following section, we fill this gap and extend this technique to mixed states.

Gramian method for mixed states
In the NISQ era, it is essential to have bounds which are valid for mixed states as these appear in noisy scenarios. Here, we build on the Gramian method and derive bounds which are valid for mixed states and which, in contrast to Theorem 1, take into account the second moment of the observable of interest. In principle, as more information is included, one can expect that this results in a tighter bound at the cost of having to measure additionally the expectation value of A 2 . The following theorem provides a lower bound to expectation values of non-negative observables: In the case where the target state σ is an eigenstate of an observable A, the Gramian method allows to derive a further bound. While the assumptions here are stronger, this bound is particularly useful in a applications such as the variational quantum eigensolver and when the observable of interest commutes with a Hamiltonian H for which the target state is an eigenstate. Formally, we have the following result: Theorem 3 (Eigenvalues). Let ρ ∈ S(H d ) be a density operator and let A be an arbitrary observable with eigenstate ψ⟩ and eigenvalue λ, A ψ⟩ = λ ψ⟩. Suppose that ≥ 0 is such that F(ρ, ψ⟩) = ⟨ψ ρ ψ⟩ ≥ 1 − . Then, lower and upper bounds for λ can be expressed as The proofs for both Theorems presented in this section take as starting the point Eq. (7) which is first extended to mixed state via purifications and Uhlmann's Theorem. The second step of the proof is a rewriting of the inequality in the case where the target state is an eigenstate of A (Theorem 3), and, in the case of Theorem 2, an application of the Cauchy-Schwarz inequality. The proofs are given in Appendix B and Appendix C.

E. Comparison of the bounds
We have seen three different methods to derive robustness interval. Namely, the interval based on SDP given in Theorem 1, the expectation value lower bound based on the Gramian method from Theorem 2, and the robustness interval for eigenvalues from Theorem 3, which is also based on the Gramian method. As a first observation, we notice that the SDP bounds are dependent only on the first moment of the observable, while the bounds derived from the Gramian method take into account the second moment via the variance. In principle, this hints at a trade-off between accuracy and efficiency. That is, by taking into account higher moments, which comes at a higher computational cost, one can hope for an improvement in accuracy as more information is included. On the other hand, the SDP bounds can be calculated as a postprocessing step and thus do not require to measure additional statistics. However, as less information is included, this typically comes at the cost of lower accuracy.
On the practical side, one needs to consider that for the SDP bounds to be applicable, it is required that the observable lies between −1 and 1. In practice, however, this is not always the case and the observable needs to be appropriately rescaled, e.g. by using its eigenvalues. As the exact eigenvalues might not be available, one needs to use lower and upper bounds on these, which results in a loss in tightness. This is because the set of feasible points in the SDP from Eq. (3) becomes larger and hence loosens the bounds. A similar issue emerges in the lower bound for expectation values based on the Gramian method where the observable needs to be positive semidefinite. If this assumption is violated, one again needs to apply an appropriate transformation of the observable, leading to a potentially looser bound. Instead of scaling, one can decompose the observable into individual terms, each satisfying the constraints, and then bound each term separately and aggregate the bounds over the decomposition. In Sec. III we consider such a decomposition in the context of VQE. Specifically, we decompose the underlying Hamiltonian into groups of mutually commuting Pauli terms and bound the expectation of each group separately. In contrast, the eigenvalue bound based on the Gramian method does not suffer from these issues and it is applicable for general observables. It is worth remarking that this comes at the cost of less generality in the sense that the bound only applies to eigenvalues rather than general expectation values.
Assuming that the observable A = P is a projection, satisfying P 2 = P , allows for a direct comparison between the bounds. Note that, in this case, the variance is fully determined by the first moment via (∆P ρ ) 2 = ⟨P ⟩ ρ − ⟨P ⟩ 2 ρ and hence we expect that the Gramian Expectation bound should not be tighter than the SDP bound. First, we incorporate the knowledge that P is a projection in the SDP lower bound by applying it to the observable 2P − 1 so that we have the bound which is exactly the same as the lower bound derived via the Gramian method in Theorem 2 when applied to the projection P . We can also compare this bound to the Gramian eigenvalue bound from Theorem 3. Since the latter is less general, in the sense that it only holds for target states which are eigenstates, we expect this to be tighter than the expectation value bound. As can be seen from Fig. 2, this is indeed the case. Finally, we notice that all of the above bounds are faithful in the sense that, as the approximation error vanishes → 0, the bounds converge to the true expectation value ⟨A⟩ σ . To compare the rate of convergence, consider the case of pure states with the target state given by σ = ψ⟩⟨ψ and the approximation state ρ = φ⟩⟨φ with With this, one can explicitly show that the error between each bound and the true expectation ⟨A⟩ σ scales with O( √ ) as → 0. For values of close to 1 on the other hand, we remark that both expectation value bounds tend towards the trivial bounds, namely 0 for the expectation value bound, and ±1 for the SDP bounds. This ultimately stems from the underlying assumptions required for the bounds to hold. In contrast, the Gramian eigenvalue bound has no assumptions on the observable A and the bounds diverge as approaches 1.

F. Fidelity estimation
All bounds presented so far have in common that they depend on the fidelity with the target state σ. However, Difference between the SDP bounds from Theorem 1 and the Gramian eigenvalue bound from Theorem 3 as a function of the infidelity and the expectation value ⟨P ⟩ρ. The observable is assumed to be a projection P and the target state is an eigenstate of the observable. The difference is calculated by subtracting the SDP bound from the Gramian bound. a) shows the difference between lower bounds, b) shows the difference between the upper bounds. As can be seen from the figures, the Gramian eigenvalue bound is always more accurate than the expectation bound. Note that the Gramian expectation value lower bound (Theorem 2) equals the SDP lower bound under these assumptions.
in many practical settings, it is not possible to access the target state and thus difficult to obtain even a lower bound to the true fidelity. Here we address this topic and present lower bounds on the true fidelity for the case where the target state is the ground state of a Hamiltonian H.
Let H be a Hamiltonian with eigenvalues λ 0 ≤ λ 1 ≤ . . . ≤ λ d and assume that λ 0 has geometric multiplicity 1 so that the corresponding ground state ψ 0 ⟩ is unique. Let ρ be a possibly mixed state approximation of ψ 0 ⟩. If both λ 0 and λ 1 are known, one can make use of Eckart's criterion [27]: In scenarios where knowledge of the lowest lying eigenvalues λ 0 and λ 1 is available, one can thus directly lowerbound the fidelity and use Eq. (11) in the computation of the robustness intervals. In scenarios where one does not have full knowledge of these eigenvalues (or, in the least, corresponding bounds), Eckart's criterion cannot be directly applied. However, we can still use the inequality if a lower degree of knowledge about the spectrum of H is available. If it is known that the energy estimate ⟨H⟩ ρ is closer to λ 0 than to λ 1 then, as an immediate consequence of Eckart's criterion, one finds that We remark that substituting Eq. (12) into the Gramian eigenvalue bounds from Theorem 3 yields the mixed state extension of the Weinstein bounds [28,29] in the nondegenerate case. If, in addition, a lower bound δ on the spectral gap is known such that λ 1 − λ 0 ≥ δ, then we have the bound derived in Ref. [30], which is a nontrivial lower bound whenever the variance is small enough such that ∆H ρ ≤ δ. With a similar technique, one obtains a further tightening of the bound: for variances with ∆H ρ ≤ δ 2. We note that this bound has also been reported in Refs. [31,32] and we provide an alternative proof in the Appendix. In practice, the bound δ on the spectral gap can be estimated via classical methods [33].
The above bounds hold for Hamiltonians H whose lowest eigenvalue is non-degenerate. In Appendix D we consider the degenerate case and show that when the approximate state ρ is pure, then there always exists a state ψ⟩ which is an element of the eigenspace associated with the lowest (possibly degenerate) eigenvalue, and for which the above fidelity lower bounds hold. However, if ρ is a mixed state, this cannot in general be said, as is shown in the appendix with a counterexample. In summary, when the approximate state ρ is allowed to be mixed, then the fidelity bounds are applicable only when the underlying Hamiltonian has a non-degenerate ground state. If, on the other hand, ρ is pure, then the bounds hold also in the degenerate case.
The fidelity bounds presented so far have in common that some degree of knowledge about the ground state in question is required. However, this degree of knowledge is not necessarily available in practice. One scenario in which this can potentially be circumvented is in the context of variational quantum imaginary time evolution (VarQITE). In general, VarQITE is a technique to find the ground state of a Hamiltonian H [34-36] by projecting the time evolution of the initial state to the evolution of the ansatz parameters. VarQITE typically comes with an approximation error, stemming from a limited expressibility of the Ansatz state or from noise. In Ref. [37], this approximation error is quantified in terms of an upper bound on the Bures distance between the evolved state and the true ground state. Since there is a one-to-one correspondence between Bures distance and fidelity, these error bounds can be converted to a lower bound on the latter. This in turn can then be used to calculate the eigenvalue and expectation bounds presented in this work. Thus, in this scenario, our bounds can be evaluated without requiring any knowledge on the true ground state.

III. APPLICATIONS
In this section, we put into practice the theoretical results presented in the previous sections and calculate the robustness intervals for ground state energies of electronic structure Hamiltonians when the approximation of the ground state is provided by a VQE. We remark that, while VQE serves as an example application, our results are not limited to ground state energies but can be used in a more general context where the goal is to calculate error bounds for expectation values. Consider a Hamiltonian H with Pauli decomposition and let ρ be an approximation to the true ground state ψ 0 ⟩. Given ≥ 0 such that ⟨ψ 0 ρ ψ 0 ⟩ ≥ 1 − , and an estimate of the variance ⟨H 2 ⟩ ρ − ⟨H⟩ 2 ρ , it is straightforward to evaluate the Gramian eigenvalue bounds from Theorem 3. For the expectation value bounds derived via SDP and via the Gramian method from Theorems 1 and 2), one needs to be more careful since the Hamiltonian H might violate the underlying assumptions. To evaluate the latter, we can account for this by adding a sufficiently large constant c such thatH ∶= H + c1 ≥ 0 and calculate the bound forH, before reversing the translation in order to get the desired bound for H. Clearly, a valid choice for c is given by −λ 0 where λ 0 is the lowest eigenvalue of H. However, it is not always clear which constant c leads to the tightest lower bound. Similarly, to evaluate the SDP bounds, we need to apply Theorem 1 to operators which are bounded between ±1. If the full spectrum of H was known, one could normalize H using these eigenvalues. However, in the context of VQE, the spectrum is not a priori known as this is precisely the task that VQE is designed to solve, and we need a different approach for the expectation value bounds. The idea is to partition the terms in the Pauli decomposition from Eq. (15) into groups so that each term corresponding to a group can be normalized. To this end, we first partition H into groups of mutually qubit-wise commuting terms Given such a representation, the spectrum of each of the H k can be calculated classically in order to scale H k →H k appropriately such that the assumptions for the bounds are satisfied. One can then compute the bounds for each of the terms in the summation and get the final bounds by aggregating the individual bounds. We further make use of the approach presented in Refs. [38,39] where one applies a unitary transform U k to each of the H k terms so that single-qubit measurement protocols can be used. Specifically, instead of measuring H k under the state ρ, one measures A k = U k H k U † k under the unitarily transformed U k ρU † k . One can then scale each A k appropriately by classically computing its eigenvalues and apply the expectation value bounds (Theorems 1 and 2) to each term separately before aggregating.

A. Numerical simulations
Here, we numerically investigate the different robustness bounds for the ground state energies for a set of electronic structure Hamiltonians, namely H 2 , LiH and BeH 2 molecules where the qubit Hamiltonians are obtained within the basis-set-free approach of Ref. [14] using directly determined pair-natural orbitals on MP2 level [15]. All our experiments have been implemented in the Tequila [13] library using the qubit encodings from openfermion [40], optimizers from scipy [41], mad-ness [42] as chemistry backend, qulacs [43] as the simulation backend for noiseless simulations and qiskit [44] for simulations which include noise. We model noise as a combination of bitflip channels acting on single qubit gates with error probability 0.01, and depolarizing noise acting on two qubit gates, also with an error probability of 1%.
For a given Hamiltonian H, we first approximate its ground state ψ 0 ⟩ via VQE. That is, for an Ansatz state ρ θ with parameters θ one minimizes the objective ⟨H⟩ ρ θ and obtains optimal parameters θ * = arg min θ ⟨H⟩ ρ θ . It follows from the Rayleigh-Ritz variational principle [45,46] that the expectation ⟨H⟩ ρ * θ is an upper bound to the true ground state energy λ 0 . The such obtained ground state approximation ρ θ * is then used to evaluate the bounds by computing the relevant statistics, i.e. expectation values and variances of observables under this state. We notice that the quality of this state in terms of a distance to the true ground state is not easily obtainable without having some prior knowledge over the system of interest (see also Sec. II F in this regard). For this reason and in order to investigate and compare the bounds, here we assume knowledge of the true fidelity with the ground state. In practice, this is of course not realistic and, as discussed previously, one needs to approximate the true fidelity. Given the ground state approximation ρ θ * and the fidelity F(ρ θ * , ψ 0 ⟩⟨ψ 0 ), we then estimate the expectation values and variances under ρ θ * in order to evaluate the bounds. In the noiseless scenario, these statistics can be calculated exactly, whereas in the noisy scenario they need to be estimated due to finite sampling errors. Thus, in the noisy case, we repeat the calculation of the bounds 20 times and report one-sided 99%-confidence intervals.
In Fig. 3, we consider the noisy scenario and compare the different types of bounds for H 2 (2, 4) and LiH(2, 4) with approximation states provided by VQE with an UpCCGSD Ansatz [16] and optimized fermionic gradients [47]. For both molecules, we notice that the Gramian eigenvalue bound is the tightest, while the expectation value bounds are less tight. However, this is not surprising, as the eigenvalue bound is more suited for this task, compared to the other bounds which hold more generally for expectation values. In Fig. 4, we again consider the noisy scenario and compare the Gramian eigenvalue bounds for approximation states obtained via the SPA Ansatz [48] and via the UpCCGSD Ansatz for H 2 (2, 4), LiH(2, 4) and BeH 2 (4,8). We first notice that the SPA Ansatz is generally less vulnerable to noise, which stems from the associated shallow circuits, compared to the UpCCGSD Ansatz. In particular, SPA and UpCCGSD have the same expressibility for H 2 but, since SPA uses more efficient compiling, its energy estimates and lower bounds are more accurate compared to UpCCGSD. A further observation to be made is that in most cases, the lower bound is closer to the true ground state energy than the corresponding VQE, as can be seen from the error plots. The exception here   Theorem 3) for LiH (2,4)and BeH2 (4,8). The lower half in each figure shows the error between VQE and the true energy E0 computed as V QE − E0, and the error between E0 and the lower bound (LB) given by E0−LB. Here, an ideal scenario without noise is simulated and the approximation errors stem from the limited expressibility of the Ansatz state.
is BeH 2 with an UpCCGSD Ansatz where the noise level is too high for VQE to converge to a good enough approximation. Indeed, the ground state fidelities for different bond lengths vary between 0.01 and 0.1 for UpCCGSD and between 0.37 and 0.92 for the SPA Ansatz. In Appendix E we show robustness intervals for the LiH(2, 4) molecule with the error rate increased to 10%. For the SPA ansatz, even with this error rate, the ground state fidelities vary between 0.51 and 0.65 while the UpCCGSD states have low ground state fidelities in the range 0.1. In other words, UpCCGSD fails to converge to states which are close to the true ground state. It is interesting to note that for lowest ground state fidelities, the expectation value bounds reduce to trivial bounds and the eigenvalue bound starts to diverge. In Fig. 5 we consider the noiseless scenario with an SPA ansatz for LiH and BeH 2 . Here we again see that in both cases the lower bound is closer to the true value than VQE, in some regions even reaching chemical accuracy where VQE does not. This is interesting, as it indicates that even though for Ansätze which lack the expressive power required to approximate the ground state, it is still possible to obtain an estimate of the ground state energy within chemical accuracy by calculating the lower bound. It is worth remarking that these bounds are obtained under the assumption of having complete knowledge of the true ground state fidelity, an assumption which is idealistic and typically violated in practice. Nevertheless, these results indicate that, in principle, it is possible to obtain very accurate lower bounds to ground state energies of electronic structure Hamiltonians.

B. Implementation
All our robustness intervals are implemented in the open source Tequila [13] library. In the following example, we run VQE for the H 2 Hamiltonian in a minimal representation (4 qubits), before computing the lower and upper bounds based on the optimized circuit, using the function robustness_interval: Used in this way, the function calculates the robustness interval for all three methods and returns the tightest bounds. Alternatively, one can specify the type of bound via the keywords kind and method where the former stands for which kind of interval is desired, that is expectation or eigenvalue, and the latter stands for the method used to obtain the bound (Gramian or SDP). For example, calculating a robustness interval for eigenvalues using the Gramian method, can be implemented as robustness_interval(..., kind="eigenvalue", method="gramian"). In general, any type of expectation value can be used. Note that our implementation is agnostic with respect to the molecular representation, so that replacing n_pno=1 with basis_set="sto-3g" will lead to a 4 qubit Hamiltonian in a traditional basis set.

IV. DISCUSSION AND CONLUSIONS
The current experimental stage of quantum computation offers the possibility to explore the physical and chemical properties of small systems and novel quantum algorithms are being developed to extract the most from this first generation of quantum devices. However, this potential for computational advantage, compared to classical methods, comes at a price of noisy and imperfect simulations stemming from low qubit counts and thus the lack of quantum error correcting qubits. The Variational Quantum Eigensolver (VQE) is the canonical example of these NISQ algorithms that allow us to obtain an approximation of Hamiltonian eigenstates by exploiting the variational principle of quantum mechanics. Besides the broad applications and promising results of this approach [3,30,49], its performance guarantees should be studied and understood.
In this work we have made first progress in this direction and have derived robustness intervals for quantum measurements of expectation values. For a target state σ, these intervals are guaranteed to contain the true expectation value ⟨A⟩ σ of an observable A when we only have access to an approximation ρ. Based on resource constraints, accuracy requirements, and depending on the task, we have seen three different types of robustness intervals. Firstly, based on its formulation as an SDP, we have derived upper and lower bounds to ⟨A⟩ σ which take into account only the first moment of the observable A and can thus be obtained by post-processing measurements of ⟨A⟩ ρ together with the fidelity with the target state F(ρ, σ). Secondly, to take into account further moments of A and get tighter bounds, we have revisited the Gramian method [12] and extended this technique to mixed states, thereby enabling their applicability in noisy scenarios which are prevalent in the NISQ era. This has led to a further lower bound to expectation values and, additionally, to lower and upper bounds on eigenvalues of a given Hamiltonian. We have also implemented our bounds in the open source Tequila [13] library. Furthermore, we have validated our results with numerical simulations of noisy and noiseless scenarios with VQE as an example application to calculate robustness intervals for ground state energies of electronic structure Hamiltonians of H 2 , LiH and BeH 2 . Within these experiments, we observed that the lower bound on the ground state energy is up to one order of magnitude closer to the true ground state energy compared to the respective VQE estimate, at times even reaching chemical accuracy where VQE does not. These results are promising as they indicate that our bounds not only provide sound guarantees for the performance of these algorithms, but can even be used in conjunction and deliver further accuracy im-provements.
The main requirement of the bounds obtained is the knowledge of the fidelity between the target state and its approximation. Although such a quantity is not always experimentally accessible and hence poses a challenge in the practical applicability of these bounds, there exist algorithms, such as within the variational quantum imaginary time evolution [37] framework, which allow for a quantification of the required approximation error in terms of distances between the target and the approximate state. Nonetheless, our aim is to provide a formal framework to study the robustness of broadly used approaches as are the Variational Quantum Algorithms. There are still many questions around the applicability of these quantum algorithms and its robustness against noise. Within this work, we seek to unravel the uncertainties around these state-of-the-art quantum algorithms with the goal of improving its performance and applicability.

CODE AVAILABILITY
The code for the numerical simulations is publicly available at https://github.com/DS3Lab/ robustness-intervals-quantum-measurements.
Appendix A: Proof of Theorem 1

Auxiliary Lemma
Here we state a result from quantum hypothesis testing which serves as a central tool for deriving the bounds in Theorem 1. We remark that this has been shown in the appendix of Ref. [17], although in a slightly different form and for pure states. For completeness, we include the proof here and refer the reader to [17] for proofs of specific lemmas. Recall that the semidefinite program for the optimal type-II error probability in binary quantum hypothesis testing is given by The following lemma provides a lower bound to this SDP which, in the case of pure states is saturated.
Proof. It is well known that solutions to the SDP β * are attained by Helstrom operators, i.e. operators of the form Λ t = P t,+ +X t with 0 ≤ X t ≤ P t,0 where P t,0 = 1−P t,+ −P t,− and P t,± are projections onto the null, positive and negative eigenspaces of the operator σ − tρ for t ≥ 0. In particular, optimality of the Helstrom operators follows from the fact that for any 0 ≤ λ ≤ 1 we have For a formal proof of this, see Lemma 6 in the supplementary information of [17]. The idea is now to choose t and, correspondingly, X t such that Tr[(P t,+ +X t )ρ] = α 0 which, by the above implication yields β * (α 0 ; ρ, σ) = Tr[(1−Λ t )σ]. The next step is then to show that Tr[(1 − Λ t )σ] is equal to the RHS in Eq. (A2) for pure states. In the last step, this is extended to an inequality for mixed states using Uhlmann's Theorem and the monotonicity of β * under the partial trace.
In the sequel, define t 0 to be the nonnegative number This is well defined since the function t ↦ Tr[P t,+ σ] is non-increasing and continuous from the right (see Lemmas 2 & 3 in the supplementary information of [17] for a proof of this). From Lemma 5 in the supplementary information of [17] we have the following sandwich inequality which enables us to set Λ t0 ∶= P t0,+ + q 0 ⋅ P t0,0 with so that Tr[Λ t0 ρ] = α 0 as desired. It follows that β * (α 0 ; ρ, σ) = Tr[(1 − Λ t0 )σ] and it remains to show that We start with pure states and extend to mixed states in the subsequent step.
Theorem 1 (restated). Let σ, ρ ∈ S(H d ) be density operators with F(ρ, σ) ≥ 1 − for some ≥ 0. Let A be an observable with −1 ≤ A ≤ 1 and with expectation value ⟨A⟩ ρ under ρ. For ≤ 1 2 (1 + ⟨A⟩ ρ ), the lower bound of ⟨A⟩ σ can be expressed as Similarly, for ≤ 1 2 (1 − ⟨A⟩ ρ ), the upper bound of ⟨A⟩ σ becomes Proof. The idea is to first formalize the problem as a semidefintie program which takes into account the first moment of A under ρ and the assumption that −1 ≤ A ≤ 1. This is then connected to the SDP Eq. (A1) where the second equality follows from the fact that replacing the equality with an inequality in the constraint of the SDP Eq. (A1) leads to the same solution. It then follows from Lemma 1 that and again use Lemma 1 and obtain the lower bound

Tightness of the SDP bound
Here we show that the SDP bounds presented in Theorem 1 are tight for pure states in the sense that, under the given assumptions, there exists an observable which saturates the bound in the case that F(ρ, σ) = 1 − . Let us first consider the upper bound and recall that in the proof of Theorem 1 we have shown where the last inequality follows from Lemma 1. Additionally, it follows from Lemma 1 that for pure states this inequality is indeed an equality, that is, we have shown that for pure states we have β * (α 0 ; ρ, σ) = α 0 (2 − 1) + (1 − ) − 2 α 0 (1 − )(1 − α 0 ) (A26) for 1− ≥ α 0 for arbitrary α 0 ∈ [0, 1]. This followed from constructing an operator Λ ⋆ with Tr[σ(1−Λ ⋆ )] = β * (α 0 ; ρ, σ). Let Λ ⋆ be such an operator for α 0 = Tr[Λ ⋆ ρ] and let A ⋆ ∶= 2Λ ⋆ − 1. We notice that which shows that the bound is indeed saturated for the observable A ⋆ . Tightness of the lower bound can be shown analogously.
and similarly Replacing A with A 2 in the above, we find Finally, we have where ⟨⋅, ⋅⟩ HS denotes the Hilbert-Schmidt inner product. Without loss of generality, we assume that ⟨Φ Ψ⟩ is real and non-negative since otherwise we can multiply each purificiation by a global phase. Applying the Gramian inequalities to the observable A ⊗ 1 and the purifications Ψ⟩, Φ⟩, we find Thus, we have shown that inequalities similar to Eq. (7) also hold for mixed states. To finish the proof, note that by assumption A ≥ 0 and hence A has a square root A = A Under the condition that F(ρ, σ) (1 − F(ρ, σ)) ≥ ∆A ρ ⟨A⟩ ρ , we can square both sides of the inequality Notice that the RHS is monotonically decreasing as F(ρ, σ) decreases. Hence, we can replace the true fidelity by a lower bound to it. In particular, for ≥ 0 with F(ρ, σ) ≥ 1 − and (1 − ) ≥ ∆A ρ ⟨A⟩ ρ we get which is the desired result.

a. Pure states
Suppose that ρ is a pure state ρ = φ⟩⟨φ . For Eckart's criterion, an analogous calculation leads to and hence so that Eckart's criterion holds in the degenerate case for this particular choice of eigenstate ψ⟩ and for pure approximation states φ⟩. Using again analogous calculations, we also obtain the extensions of the bounds Eq. (13) and Eq. (14) for pure approximation state φ⟩ in the degenerate case and for the same choice of eigenstate ψ⟩.

b. Counterexample for mixed states
If the approximation state ρ is allowed to be arbitrarily mixed, the above fidelity bounds do not hold in general. Indeed, consider the Hamiltonian for arbitrary λ, µ ∈ R with λ < µ and some arbitrary unitary U . Furthermore, let ρ be the maximally mixed state ρ = 1 3 1 3 and note that ⟨H⟩ ρ = 2λ+µ 3 . Thus, for any ψ⟩ we find that in violation of Eckart's criterion. To see that we can also construct a counterexample for the other two bounds, we calculate the variance and notice that and similarly ⟨ψ ρ ψ⟩ = 1 3