johnjnay

Bio

A.I. researcher and co-founder and CEO of an A.I. technology company, Brooklyn Artificial Intelligence Research (Skopos Labs, Inc.), which owns the investment management firm, Brooklyn Investment Group (https://bkln.com). 

Conducted research funded by the U.S. National Science Foundation and the U.S. Office of Naval Research. Created first A.I. course at the NYU School of Law. Published research on A.I., finance, law, policy, economics, and climate change. Publications at http://johnjnay.com, and Twitter at https://twitter.com/johnjnay.

Comments
6

This was cross-posted here as well: 

A follow-up thought based on conversations catalyzed by this post:

Much of the research on governing AI and managing its potential unintended consequences currently falls into two ends of a spectrum related to assumptions of the imminence of transformative AGI. Research operating under the assumption of a high probability of near-term transformative AI (e.g., within 10-15 years) is typically focused more on how to align AGI with ideal aggregations of human preferences (through yet to be tested aggregation processes). Research operating under the assumption of a low probability of near-term transformative AI is typically focused on how to reduce discriminatory, safety, and privacy harms posed by present-day (relatively "dumb") AI systems. The proposal in this post seeks a framework that, over time, bridges these two important ends of the AI safety spectrum. 

Interesting. I will think more about the sandwiching approach between non-legal experts and legal experts.

Hi Geoffrey, thank you for this feedback.

On your background knowledge comment, I agree that is an important open question (for this proposal, and other alignment techniques).

Related to that, I have been thinking through the systematic selection of which data sets are best suited for self-supervised pre-training of large language models - an active area of research in AI capabilities and Foundation Models more generally, which may be even more important for this application to legal data. For self-supervision on legal data, we could use (at least) two filters to guide data selection and data structuring processes. 

First, is the goal of training on a data point to embed world knowledge into AI, or legal task knowledge? Learning that humans in the U.S. drive on the right side of the road is learning world knowledge; whereas, learning how to map a statute about driving rules to a new fact pattern in the real world is learning how to conduct a legal reasoning task. World knowledge can be learned from legal and non-legal corpora. Legal task knowledge can primarily be learned from legal data.

Second, is the approximate nature of the uncertainty  that an AI could theoretically resolve by training on a data point epistemic or aleatory ? If the nature of the uncertainty is epistemic – e.g., whether citizens prefer climate change risk reduction over endangered species protection – then it is fruitful to apply as much data as we can to learning functions to closer approximate the underlying fact about the world or about law. If the nature of the uncertainty is more of an aleatory flavor – e.g., the middle name of the defendant in a case – then there is enough inherent randomness that we would seek to avoid attempting to learn anything about that fact or data point. 

There are many other aspects of  self-supervised pre-training data curation that we will need to explore, but figured I'd share a couple that are top of mind in the context of your world knowledge comment.

Public law informs AI more through negative than positive directives; and therefore it’s unclear the extent to which policy – outside of the human-AI “contract and standards” type of alignment we are working on – can inform which goals AI should proactively pursue to improve the world on society’s behalf.  I agree with your comment that, "law tends to track situations where humans have conflicts of interest with each other, and it might not track universal values that are so obvious to everyone that conflicts of interest hardly ever arise." This is a great illustration of the need to complement the Law Informs Code approach with other approaches to specifying human values. But I believe there are challenges with using the "AI Ethics" approach as the core framework, see section IV. PUBLIC LAW: SOCIETY-AI ALIGNMENT of the longer form version of this post, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4218031 . I think a blend of the frameworks could most fruitful.

Finally, it would be very interesting to conduct research on the possibility of "cross-cultural universals in legal systems that exemplify some common ground for human values," and which domains of law have the most cultural overlap. There are many exciting threads to pursue here!

Hi Charlie, thank you for your comment.

I  cite many of Dylan's papers in the longer form version of this post: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4218031

I will check out Xuan's talk. Thanks for sharing that.

Instead of:

Law is a computational engine that converts human values into legible directives.

I could expand the statement to cover the larger project of what we are working on:

Law and legal interpretation form a computational engine that converts human values into legible directives.

One of the primary goals of this research agenda is to teach AI to follow the spirit of the law in a human-recognizable way. This entails leveraging existing human capabilities for the "law-making" / "contract-drafting" part (how do  we use  the theory and practice of law about how to tell agents what to do?), and conducting research on building AI capabilities for the interpretation part (how do our machine learning processes use data and processes from the theory and practice of law about how agents interpret those directives / contracts?). 

Reinforcement learning with human attorney feedback (there are more than 1.3 million lawyers in the US) via natural language interactions with AI models is potentially a powerful process to teach (through training, or fine-tuning, or extraction of templates for in-context prompting of large language models) statutory interpretation, argumentation, and case-based reasoning, which can then be applied more broadly for AI alignment. Models could be trained to assist human attorney evaluators, which theoretically, in partnership with the humans, could allow the combined human-AI evaluation team to have capabilities that surpass the legal understanding of the expert humans alone. 

The Foundation Models in use today, e.g., GPT-3, have, effectively, conducted a form of behavioral cloning on a large portion of the Internet to leverage billions of human actions (through natural language expressions). It may be possible to, similarly, leverage billions of human legal data points to build Law Foundation Models through large-scale language model self-supervision on pre-processed legal text data. 

Aspects of legal standards, and the "spirit" of the law, can be learned directly from legal data.  We could also codify examples of human and corporate behavior exhibiting standards such as fiduciary duty into a structured format to evaluate the standards-understanding capabilities of AI models.  The legal data available for AI systems to learn from, or be evaluated on, includes textual data from all types of law (constitutional, statutory, administrative, case, and contractual),  legal training tools (e.g., bar exam outlines, casebooks, and software for teaching the casuistic approach), rule-based legal reasoning programs,  and human-in-the-loop live feedback from law and policy human experts.  The latter two could simulate state-action-reward spaces for AI fine-tuning or validation, and the former could be processed to do so.

Automated data curation processes to convert textual legal data into either state-action-reward tuples, or contextual constraints for shaping candidate action choices conditional on the state, is an important frontier in this research agenda (and promising for application to case law text data, contracts, and legal training materials). General AI capabilities research has recently found that learning from textual descriptions, rather than direct instruction, may allow models to learn reward functions that better generalize. Fortunately, much of law is embedded more in the form of descriptions and standards than it is in the form of direct instructions and specific rules. Descriptions of the application of standards provides a rich and large surface area to learn from.

Textual data can be curated and labeled for these purposes. We will aim for two outcomes with this labeling. First, data that can be used to evaluate how well AI models understand legal standards. Second, the possibility that the initial “gold-standard” human expert labeled data can be used to generate additional much larger sets of data through automated curation and processing of full corpora of legal text, and through model interaction with human feedback.

I think your statement:

"This research direction does not look like a Bold New Way to do AI alignment, instead it looks like a Somewhat Bold New Way to apply AI alignment work that is fully contiguous with other alignment research" 

is spot on. That is how I was thinking about it, but I should have made that more clear; perhaps I should work on a follow-up post at some point that explicitly explores the intersections of Law Informs Code with other strands of alignment research. Some of this is in the longer form version of this post, but with this inspiration from you, I may try to go further in that direction (although I am already beyond the length the Journal editors want!).

That's awesome - thank you for sharing!

 

Would love to chat as well.