Once again this semester, LII hosted a team of M.Eng. students, who explored the application of human language technologies to legal information retrieval tasks. Under the guidance of LII’s Language and Data Scientist Dr. Sylvia Kwakye and recent M.Eng. graduate Adrian Hilton, the students began with work-in-progress and refined data, techniques, and visualizations to get the data ready for the public to use on the LII website.

The students met weekly, sharing the results of their work, comparing notes on techniques and challenges, and reaching consensus on what looked most promising to try next. As they worked, they encountered — and became adept at untangling — difficulties with source datasets and pretrained models, alongside the usual software engineering challenges they were more accustomed to encountering from their other coursework.

There was broad agreement that this experience would be particularly relevant to the work they are about to take on at their first jobs in the tech world. These days it seems as though every week there’s a new language model, approach, or tool to try out, and nearly as often, a new flurry of headlines about less-than-credible results from a big technology company’s latest product or feature. The ability to reality check — efficiently — has never been more important, and this semester’s project gave the students opportunities to apply their considerable creativity to answering the question “are we there yet?”.

As a bonus, the students’ in-depth exploration of alignments between a general legal ontology, corpus-specific topical indexes, and a topic model derived directly from the language of legal texts provides a strong foundation on the data side for systematizing topical organization and evolving search for LII’s original content, including the Wex legal dictionary / encyclopedia, which was the focus of LII’s first hackathon.

We held our first hackathon on Sunday March 24, 2024. Students from various schools came together at Cornell Law School for the event, forming five interdisciplinary teams to answer the question: How would you organize the content in Wex, our free legal dictionary and encyclopedia?

Before they arrived in Myron Taylor Hall on Sunday morning, we provided the students with a “Welcome Packet” containing a brief history and description of both LII in general and the Wex collection in specific. During the hackathon event, the teams had just over 6 hours to work together to prepare and propose their answer. All five teams came up with incredibly creative and unique ways to organize the content in Wex and to make the content more accessible to users. While it was difficult for judges to choose an overall winner–and all teams won a prize–our panel of judges ultimately awarded the grand prize to Team 5 – “ALIIGN.” Team ALIIGN consisted of Anurag Koyyada – a JD student from Cornell Law School, Jasmine Li – a Computer Science and Philosophy major from the College of Engineering, and Adler Weber & Christopher Price, both Computer Science majors and from the College of Arts & Sciences. ALIIGN was an initiative to streamline the Wex organization to:

• Reorganize Wex Browse, replacing the traditional alphabetical searching system with a content tag and metadata tag-based tree hierarchy
• Enhance the “Wex Search” capability – replacing our current CMS with a robust, fast ad-hoc vectorized search, and
• Modernize the user interface.

All teams made wonderful pitches to our panel of judges: Cornell Law School’s Edward Cornell Law Librarian Kim Nayyer; Vanderbilt AI Law Lab Co-Director Mark Williams; Law Library of Congress Senior Legal Information Specialist Jennifer González; Adjunct Professor of Law, Managing Director of Justia’s Verdict, and Editor-in-Chief of the Oyez Project David Kemp: and LII’s own application programmer Matt Carey. We are grateful to our contestants and judges alike for their intellect and inspiration.

The prizes and event were funded by generous gifts from Barbara Lewis, B.S. ’65, M.A. ’67, and Jack Lewis, J.D. ’69, as well as Justia.

Images and video by Alexandra Bayer and Paul Newman / Cornell University.

A year ago at this time, ChatGPT had just launched. As the timing worked out, this was not long after we’d cleaned up the crumbs from our 30th birthday cake, and at that point we couldn’t help but think that if we’d needed any reminder to keep focusing on the future, it would have been hard to find a better one. In the year since, we’ve put a dizzying array of new technologies through their paces and been reminded of both the learning curve for new tools and the magic of gaining leverage on a problem by finding the right fit between a new tool and our central task: helping people find and understand the law.  

As we’ve been talking about in several recent posts, this year has brought us an occasion to look anew at how people from all walks of life use the LII website – including (as we noted in our previous post) whether or not they ever experience our website directly at all.  In part because we have that luxury, we choose to see consumption of our website for purposes of re-use as a force multiplier – quantitatively an existential game-changer, but not all that different qualitatively from what happens when individuals use content from our website to develop training manuals for public servants, or include it in source packets for law school classes, or screenshot it for social media. 

And so far we’re finding that this particular juncture in the history of artificial intelligence increases the need for stable, reliable information whose provenance is clear. We have seen already that the combination of text generation at negligible cost and hallucination by large language models has created a misinformation force multiplier. Although qualitatively a familiar problem (most starkly for those of us who remember that one time when a false citation to a non-existent provision of LII’s U.S. Code played a role in a national news story), the asymmetry of effort required to verify information makes the proliferation of unreliable text an existential problem not just for us but for the web itself.  

Our aim is to keep the legal information we publish free, accessible, stable, and reliable – available to everyone, fact checkers included, from a reputable source at the click of a button. As we navigate this rapidly evolving information environment and find the niches where we can make unique contributions for the benefit of the public, we hope to seize opportunities to evolve while continuing to offer services the public has come to rely on.

Today is the winter solstice.  We know that because we asked Google. 

As you can see, Google didn’t return a list of web pages from which we could extract an answer, but the actual answer itself: December 21, 2023. It was then and there that I knew what I wanted to say in this blog post.

We’re all accustomed to these Featured Snippets (as Google now calls them), but when they started almost a decade ago, they were a really big deal. On one hand, it was neat to see Google use our website to, for example, answer a holiday-themed legal question:

On the other hand, getting the answer straight from Google wasn’t obviously a good thing for projects like ours. If searchers didn’t click through on that link, then we wouldn’t have an ever-growing amount of traffic to demonstrate our usefulness. We also couldn’t expect the same level of advertising revenues and user donations that made our work possible if folks stopped needing to visit our website to find the answer they were looking for. It was, quite literally, an existential crisis in the making. Despite these concerns, our traffic continued to grow steadily over the last decade to a point where continuing to use it as a leading indicator of our impacts now seems futile (a topic we touched on in another recent post).  

We’ve also written recently about the fascinating rift that’s developing in the global online Free Law world all these years later between those who welcome the bots/scrapers/crawlers that harvest content like ours to power AI systems and other future technologies and those who are fighting to keep their websites, in essence, read only. In short, what’s coming down the road makes Google’s Featured Snippets seem utterly quaint, and existential crises abound for all sorts of knowledge workers.

Both that Featured Snippets example and this recital of the issues they implicate are admittedly facile; but, we  wanted to reiterate LII’s commitment that we will continue to develop, test, refine, implement, and share innovative techniques for making legal information more findable and comprehensible for the public–whether you ultimately end up on our website or not. 

We’d like to share a bit about how we approach the heavy traffic that’s been keeping us on our toes. We’ve been talking a lot recently about the ways in which changes to Google’s traffic reports have been an occasion for us to revisit our thinking about who uses the site and how they do so. Here we’re relying not just on the data from people whose use of the website shows up in Google Analytics, but also from the ever-growing assemblage of automated user agents ranging from long-standing search engine crawlers to newer corporate large language model crawlers to custom crawlers that seem to represent some very curious researchers on home internet connections. 

A little over a month ago, LII’s leadership attended the Law Via the Internet Conference, meeting with other members of the worldwide Free Access to Law Movement. As you might guess, everyone was talking about the use and refinement of artificial intelligence tools, particularly large language models. The atmosphere encompassed a mix of excitement, caution, curiosity, and realism. But one element of the discussion stood out as a point of strong divergence: what are the limits of the “free” and “access” in “free access to law”? Does free mean “free as in beer” or “free as in speech”?

It turns out that we have a different intuition from some of our sibling LIIs who run services designed to be used primarily by members of the legal profession. Although (like us) they would never stand in the way of others gaining free access to information published by their respective governments, (also like us) they have invested significant amounts of resources over many years to standardize and enrich the documents they work with. But because (unlike us) many of them focus primarily on publishing case law, privacy concerns alone have given them plenty of reasons not to allow crawlers access to their websites. A policy not to now open their data for free AI-related consumption is not far from their long-standing practice.  Use of their services is free as in beer. 

We take a very different approach. Our content is findable primarily because it has always been crawlable, and (almost all) reusable. That policy enabled our site to reach the top of search rankings for vast numbers of search terms. By extension, it enabled our data to become part of the pool of documents used to train most of the large language models.  In turn, that openness – and the openness of other U.S.-based free access to law operations – means that even if they’re still having some trouble getting their facts straight today, the large language models we interact with have access to the data they need to get it right tomorrow. Even if we don’t ever see the readers who benefit from that data, it means that our work can serve the public in ways we can’t begin to imagine. As best we can, we offer services that are free as in speech.

But it gets tricky. These crawlers all receive free access to the LII website and in turn make LII’s work usable by millions more people than we would ever reach on our own. But their traffic is not free for us to field.  (If you’re wondering why we don’t consider changing our approach, the reality is that we are a very small group, and even if we wished to change our policy to include a paid service for corporate reusers, it would come at the expense of using our technical and administrative resources for direct support of our core mission.) For both principled and practical reasons, everyone – human or crawler alike – receives a standard of access that supports all of our users.

We recently wrote about how changes in the way Google reports traffic to websites like law.cornell.edu have given us the opportunity to revisit how we think about, and talk about, traffic. When we sat down last week to sift through the 7.4 million website sessions Google Analytics recorded for us in November, one of the first things we did was go to the “heaviest” of traffic days last month–November 29–to look for interesting use cases. What emerged amused us, and we thought you might enjoy hearing about it.  

There weren’t any particularly compelling news headlines like a high-profile indictment, an important Supreme Court decision, or a political event of the type that often drive extra folks to our website. There was also no evidence that all the folks who use our site as part of their work in legal services, or any of the myriad regulated industries, or all levels of government, etc. were particularly active on a random Wednesday six days after Thanksgiving.  

So, what was it? All signs point to law students flocking to the website to study for their final examinations. All that week, but especially that day, traffic in the Federal Rules of Civil Procedure was busier than usual, and parts of Title 28 of U S Code like Sections 1332 (diversity jurisdiction), 1367 (supplemental jurisdiction), 1391 (venue), & 1446 (removal) were far busier than usual. Anyone with a law degree will recognize those topics as the fundamental concepts addressed in any good CivPro exam.  

We always get a lot of mileage, especially within our home institution of Cornell Law School, in recounting all the ways we employ law students and expose them to the intersection of law and technology. But we don’t really talk too much about students as a segment of our user population. But they’re very much present, and sometimes we hear from them and those who teach them:

Thank you for providing much needed information for us law students and for the public at large!

A law student in Georgia

I have so much appreciation for what you do! LII is the first resource I turn to when I need to look at statutory text or regulations.

A law student in Washington, DC

I teach business law and find your site to be extremely helpful!

A law professor in Georgia

Most of my students go into solo or small firm practice. Having this resource is a life-saver for many of them because they cannot afford the various paid research services. But they still have clients who deserve justice.

A law professor in Texas

This website has been essential for my law school education!

A law student in Massachusetts

This was an invaluable resource when I was teaching, and I still consult it in retirement.

A retired law professor in Washington

I honestly do not know how I would be able to complete my weekly discussion posts and assignments without [LII].

An undergraduate legal studies major in Oregon

This website saved my [keester] more times than I care to admit in Law School and is an extraordinary research tool in the real world.

A lawyer in New York, NY (of course) 

With any luck, we’ll keep saving the [keesters] of students, lawyers, and millions of others for many years to come.