Podcast Ep #126: Legal Quality Assurance Fails: Lessons from a Real Appellate Case [AI ROI Part 2]

John: I'm going to lead with a hot take today. I don't think we're in a world where AI is creating new quality problems in legal, so much as it is exposing flaws in our quality assurance practices that have always been there.
​​​​​​​
What passes for QA in most legal settings is often just a subjective standard inconsistently applied. And even in an adversarial proceeding where lawyers on the other side of a problematic filing have a huge incentive to expose their opposing counsel's flaws, recent cases have shown that the errors can easily go undetected. But rather than me talking about it, today, I'm giving you about 20 minutes from a recent appellate argument that illustrates that point better than I ever could.

You're listening to The Agile Attorney Podcast, powered by GreenLine. I'm John Grant, and it is my mission to help legal professionals of all kinds build practices that are profitable, sustainable, and scalable for themselves and the communities they serve. Ready to become a more Agile Attorney? Let's go.

Hey everyone, welcome back. So, I'm doing something a little different today. I'm going to play you about 20 minutes of actual oral argument from the New York Court's Second Appellate Division. This is the intermediate appellate court of New York State.

The case is Landberg versus City of New York. It's a routine sidewalk trip and fall. But the merits of the case aren't too important for our purposes today. What matters is what happens in that room. The first lawyer you'll hear is arguing for the plaintiff appellant, and then you'll hear from lawyers representing both the property owner and the City of New York as defendant appellees.
Three attorneys, an appellate bench, arguments that have been briefed, read, prepared for, about as normal a setup as you get in appellate practice. But it does not go the way anyone expected. I'll let you decide what you think as you listen in, and I'll be back at the end to offer my thoughts.

Valerie Nelson: But there is another issue that I would like to move on to, and I'm going to ask you to address it. And that is, in preparing for this oral argument and reviewing the brief of appellant, it came to the attention of the court that the brief submitted by plaintiff cites to at least three cases that appear to be fictitious: Haza versus Quinta, Black v. Lack, and Xiang Fuji versus City of New York.

None of these cases, nor the quoted language appears to exist. And with respect to the last case I mentioned, the last purported case I mentioned, the Xiang case, the plaintiff's brief purports to quote from a case and incorrectly attributes certain language to the Court of Appeals.

Michael Sanders: For which case?

Valerie Nelson: The X I A N G. And also, and I'm I just want to tell you about some of my other concerns and then I'm going to ask you to respond.

Michael Sanders: Sure.

Valerie Nelson: There are 10 other cases cited in plaintiff's brief, some of which appear to misrepresent the law to some extent. So these are concerns that I have as a member of this court and I'd like you to address this issue.

Michael Sanders: So...

Valerie Nelson: How do you, how do you respond?

Michael Sanders: Plaintiff cited a plethora of cases. Those specific cites that your honor is referring to wasn't brought up in opposition, so I wasn't specifically prepared to speak to them. I'm not sure what the...

Valerie Nelson: Well, counsel, before you go any further, let me point out to you that rule 3.3a of the rules of professional conduct indicates that a lawyer shall not knowingly make a false statement of fact or law to a tribunal or fail to correct a false statement of material fact or law previously made to the tribunal by the lawyer. It doesn't depend on whether your adversary points out to you a possible fictitious case. It is the responsibility of the attorney submitting the brief to the court.

Now, I've just advised you that at least three cases are apparently totally fictitious. They don't exist. The purported citations don't exist for these cases.

Michael Sanders: I was only pointing out that the opposition...

Valerie Nelson: So how do you respond? You have submitted this to the court.

Michael Sanders: If there's any citations that are incorrect, I, my deepest apologies.

Hector Lasalle: Where'd you get them from?

Michael Sanders: I don't know what these cases were specifically...

Hector Lasalle: Do you want to take 10 minutes and go look them up? Should we, do you want, we give you 15 minutes to go and you could find them? You wrote this, right?

Michael Sanders: I wrote this, yes.

Hector Lasalle: You wrote this.

Michael Sanders: Yes. So if...

Hector Lasalle: You wrote this.

Michael Sanders: I wrote this.

Hector Lasalle: You're responsible for it.

Michael Sanders: I'm responsible for this and the reply.

Hector Lasalle: Okay. And every case you used, you found in a book, Westlaw or Lexus. Is that correct?

Michael Sanders: Or, or cited from a previous case. Yes.

Hector Lasalle: Okay. Do you want us to give you 15 minutes so you can tell us where you got these, 'cause counsel, you're an officer of the court.

Michael Sanders: Sure.

Hector Lasalle: Okay?

Michael Sanders: Sure.

Hector Lasalle: You tell me, you did, you tell me you found these in a book. You found these in Lexus, Westlaw, or some other, or, or in a physical book. Is that what you're saying you got these cases?

Michael Sanders: Or, or in Westlaw or in a previous, a, I mean I may have used...

Hector Lasalle: There's, just so you know, there, because I don't want you to dig a bigger hole here.

Michael Sanders: Sure. Sure.

Hector Lasalle: You're citing principles that don't exist. So...

Michael Sanders: I mean, I'm just, I'm listening.

Hector Lasalle: I didn't, let me tell you something, okay? We saw this last week. I was hopeful that in preparation for today that you were going to read this and say, "Oops, we made a mistake, judge." Happens sometimes, right? People put a site that's incorrect, scrivener errors, et cetera. That's what I was hoping for.

Michael Sanders: Sure.

Hector Lasalle: We didn't get that. Should we give you some time right now to go look these cases up?

Michael Sanders: I think it's going to take me more than 15 minutes. I just, I just want to explain. I was very focused on the, the case law that I really thought was most relevant towards the, the heart of the issue.

Hector Lasalle: I understand. You understand what my, what's, what's our concern?

Michael Sanders: That there's, that there's sites in the case that don't exist, sites in my brief that don't exist.

Hector Lasalle: Would you agree that's a legitimate concern?

Michael Sanders: That's absolutely a legitimate concern.

Hector Lasalle: You know, in my professional career, I've seen a lot of things, okay? We all have. I don't want to speak for my colleagues, but I'm confident they all have. And I'm sure you have too. I think every lawyer here has seen it. Times when you put a case into Lexus or Westlaw, or in the old days when you had to open up the book, and there's a scrivener's error, right? And you're like, "Okay, it wasn't page 344, it was page, you know, 644."

Michael Sanders: Sure.

Hector Lasalle: Okay? But we can find the case because you can see where the error was made, because you know the case exists. You're able to find, it's a pain in the neck.

Michael Sanders: Sure.

Hector Lasalle: But it happens. And we've all been there. I'm sure you've been on the other side of that where a lawyer's done that to you, put it in a brief or in motion papers. It's not a big deal. It's, it's annoying, but we get past it. This is something different. This is a wholesale creation of cases.

And I'm concerned that you're citing stuff, you're citing cases that don't exist. That's a big concern. This is not me pulling one case out of a string site.

Michael Sanders: Right.

Hector Lasalle: Okay? I understand. You might not, you know, you should be, you might not have read every single case in a string site. You're suggesting the Court of Appeals said things they didn't say.

Michael Sanders: Okay, if, if I, again, I, I mean, I would have to review my papers. I think with Zhu, I think it was that was an appellate decision that actually wasn't a Court of Appeals decision that I was mistaken about. Again, like I said, I just, I was really focused on the cases, the reason why I don't know every single case is I was really focused on the cases that I most of my brief was relying on, which was...

Valerie Nelson: But counsel, some of the fabricated quotations used to support your primary argument are contrary to actual law, actually contrary to what the law states on your primary arguments. These are not, not minor side issues. These are issues that you offer, that you put before this court to support your primary arguments. It is concerning, and we're bringing this to your attention so that you may explain, if possible, how this occurred.

Michael Sanders: I mean, the only thing I could, I could think of is just me over-relying on other briefs and, uh, maybe string sites for general case law. But in terms of specifically about the issue of bricks...

Hector Lasalle: Just to interrupt you. You wrote this, right?

Michael Sanders: I wrote this.

Hector Lasalle: You're responsible for this.

Michael Sanders: I'm responsible for it. Yes, your Honor.

Hector Lasalle: You're responsible. You're not going to, you're not going to tell me some associate at the firm stuck something in there that you didn't see.

Michael Sanders: No.

Hector Lasalle: You wrote this.

Michael Sanders: Yes.

Hector Lasalle: Anything else you want to add?

Michael Sanders: I guess no, nothing further. Thank you, your Honors.

Hector Lasalle: Thank you.

Ross Friscia: Uh, good morning, your Honors, and may it please the court. Ross Friscia for the owner, uh, respondent. Here the motion court correctly awarded summary judgment.

Hector Lasalle: Can I ask you a question? Before we get to anything else.

Ross Friscia: Yes.

Hector Lasalle: You saw that these cases were, I mean, when you read your adversary's brief, are, are, do, do you actually check the cases?

Ross Friscia: Uh, yes, your Honor.

Hector Lasalle: How come you didn't bring it to our attention?

Ross Friscia: So with this particular scenario, I was more focused on the specific concern.

Hector Lasalle: He was, he was raising, he, he referenced Court of Appeals standard that doesn't exist. He was using it as, as the, as a, as a component of his argument, and you, and you didn't, and you didn't think you should bring it to our attention?

Ross Friscia: So, your honor, I actually, so I, I didn't notice in particular that the principle of law that he was citing was incorrect. Over here, I, it says, uh, in one of the cases that your honor mentioned that 7210 should be construed liberally to effectuate its remedial purpose. That's correct. Uh, is, is there another case that I should address because with that principle of law, that's...

Hector Lasalle: Counsel, we, I'm gonna give you a chance to make your argument.

Ross Friscia: I'm sorry.

Hector Lasalle: I'm gonna give you every opportunity to make your argument. I'm, but I'm befuddled. I, I honestly am. I'm absolutely, but I'm not here to, lawyers make mistakes. It's not an easy profession. I don't like, I don't want to sit here beating up on lawyers, but we rely on the bar so much in what we do.

So, the first thing that I did, I don't want to speak for my colleagues, but after seeing what he wrote, when I went to your papers, I expected to see something referencing it. Not just, it wasn't one case counsel, it was several cases, and you didn't, and you didn't see fit to bring it to our attention either. It's just striking to me. I'll let you make your argument.

Ross Friscia: Yes, so I, your honor, I apologize to the court. I will do further due diligence, um, going forward from this point on.

Hector Lasalle: I hope so. You should apologize to your client, not to me.

Ross Friscia: Yes. I, I, I apologize for that and I will, um, going forward check every single case even if it stands for, you know, general principles of law like the construed liberally to effectuate remedial purpose and, and things like that. And I will bring them to the court's attention.

Valerie Nelson: Counsel, the misrepresentations here are of such a degree that they could not merely reflect a difference of opinion. As an appellate court attorney, you would have to, if you were doing the work and reading the briefs and responding to the briefs, you would, you would have to notice that something in the wording of the main brief for the appellant was wrong, if not many things being wrong.

It's concerning because we are all officers of the court, and there is a responsibility that you also have to notify the court, to do the work, notify the court when these types of misrepresentations and fictitious cases and fictitious citations and misrepresenting the holding of a Court of Appeals case, and I could go on and on. But if you read the brief and looked at the cases, you would have realized it was your responsibility also to alert the court. How do you respond?

Ross Friscia: Your honor, what I can say is we write these briefs normally to be very, very focused on the issues here, and that is a big issue, and going forward I will, you know, make sure to check every single case that is cited, even Court of Appeals cases. But when I write the reply brief, I usually tailor it to be very specific to the dispositive issue, um, here. The propositions that were cited, um, or mis-misattributed to certain Court of Appeals cases, um, did not immediately raise a red flag as I was writing, um, my brief.

What raised red flags was the application of the law as it, um, applied to appellate precedent in this department, which, um, concerns what a tree well is. And, um, you know, in that search, I relied on one case, Ivry v. City of New York, that is, I think, dispositive here, that, you know, shows a picture of a brick that is nearly identical to the brick at issue in this case. And that's really the dispositive issue.

As far as, I, I see here the Court of Appeals on page 14 of the appellant's brief, 7210 should be construed liberally to effectuate its remedial, uh, purpose. I think that's a generally accepted principle of law and quite frankly, that's why I missed it. So again, my apologies to your Honors. I will be, uh, more diligent in the future. Um, and I, if there's, uh, anything else on that point, um, that I can answer, I'd be happy to. Um, but again, I will be more diligent going forward. I hope your Honors can accept that from me.

Um, but in terms of the actual, uh, dispositive issue here, which is, um, again, the cobblestones that are, um, surrounding the tree well. Again, page 78 of the record shows a picture of where the plaintiff identified at her deposition, um, exactly where she fell. And she circled the brick closest to the tree, clearly within a flag of sidewalk that was taken out. Um, and there's dirt and there's cobblestones.

And here the city admitted to installing that. On those two points alone, I believe that that's enough to, uh, affirm the lower court's decision. And unless there are any further questions from the court, um, I'll rely on my brief.

Hector Lasalle: Thank you for your time.

Ross Friscia: Thank you.

Elizabeth Freedman: Good morning, your honors, may it please the court. Elizabeth Freedman, I'm appearing for respondent City of New York. Your Honor, um, in appellant's brief point three, uh, point three addresses the issue of the prior written notice, and that was the point of the brief that I, that we in my office and I focused on.

Valerie Nelson: But of course, you read the entire brief of the appellant, is that correct?

Elizabeth Freedman: Yes, your Honor.

Valerie Nelson: So how do you explain your failure to bring to the attention of this court that a brief was filed with this court by appellant's counsel with apparent fabrications and misrepresentations?

Elizabeth Freedman: Your Honor, the point of the brief dealing with the owner's liability, we didn't take a position on the owner's liability. We didn't, and, and I certainly read the briefs. I certainly read all of the briefs here, but I certainly didn't focus on it because it was not our issue. And I do apologize to the court for not catching that, but I tended to focus more on the issue of prior written notice. In that point of the brief, the plaintiff cited two cases. Neither one of them raised red flags.

Those two cases are Paraznik, um, where it says that an exception exists where the municipality affirmatively created the defective condition. That seemed to be accurate. That didn't raise any red flags. And the only other case they cited in that point is Rogers versus City of New York, stating that a two-year search is sufficient and can be sufficient. In our brief, I tried to state the law correctly.

Valerie Nelson: Counsel, I, I suggest that at some point in the very near future, you return to your review of Rogers v. City of New York because I have some concerns that that is a case that appears, as it appears in the appellant's brief, it appears to misrepresent the law to some extent.

Elizabeth Freedman: Your Honor, so in that case, uh, the...

Valerie Nelson: So I, I would, I would tread very lightly in discussing with this court that you feel that the reference in appellant's brief to Rogers v. City of New York was appropriate.

Elizabeth Freedman: Well...

Valerie Nelson: I don't agree with that.

Elizabeth Freedman: Okay. Uh, I apologize, your honor. Um, it seemed the part I was reading, it held that a two-year search can be sufficient. Um, and I basically cited many cases in our brief that basically stated that a two-year search is sufficient. So I tried to state the, uh, the, uh, um, a principle of law here accurately. Um, and they, uh, I basically felt that it was just simply a statement of the law about a two-year search.

And I did not delve into that particular case to see what the holding was since it didn't seem that that was uh, that the plaintiff was relying on that other than to state that a two-year search can be adequate, but that they're basically stating that, held that a two-year search can be sufficient, then goes on to say that holding is not a blanket rule applicable in all circumstances. Um, so I felt that was the opinion of counsel and not the holding of the court in that case.

But in our brief, we did cite cases that do unequivocally from this court, unequivocally state that a two-year search for records is sufficient and that here, um, the city demonstrated the lack of prior written notice and that the plaintiff in opposition did not create an issue of fact as to whether or not the city affirmatively created the condition through an affirmative act of negligence. The undisputed evidence from the building superintendent was that the bricks were even when they were installed and, and, uh, that they, that they became uneven over time.

That is certainly not affirmative negligence here. And, um, again, your honor, I apologize. Um, I, uh, tried to state the law accurately in our brief and I apologize for not looking at the entire panoply of cases cited in the, uh, brief since that was not the focus of our particular issue in this case. And we do feel that summary judgment was correctly granted as, um, and dismissing the case as, again, as the complaint is against the City of New York for the reasons that the court, the Supreme Court stated in its decision. And unless the court has any further questions? We rest on our brief. Thank you, your Honor.

Hector Lasalle: Thank you. Well, don't go anywhere yet. Have a seat. Everybody please have a seat. I just want to say this to, to you all. This is a very distressing situation for the court. I know this is an outlier. We're very fortunate. My colleagues and I, we, we have the privilege of working with what I think is one of the best benches in the state, the bars in the state, pardon me. The appellate bar, uh, here in the City of New York and its surrounding suburbs. Uh, we see excellent work.

For me, you know, personally, it's been the highlight of my career to have the opportunity to work with such outstanding judges and to have the opportunity to work with such outstanding lawyers. But part of this profession, a big component of it is that an element of trust. And mistakes are made. We make mistakes as judges. We've made mistakes. I don't want to speak for my colleagues, but I dare say that we've all made mistakes as practitioners.

And we work very hard when there are, uh, mistakes to try to give the benefit of the doubt to those lawyers who practice before us. We know how difficult your respective jobs are. And in reviewing this, I know my colleagues and I have tried to give every benefit of the doubt to the lawyers before us. But the work, the lawyering that was done here, at least as far as the briefs and specifically the promulgating of false cases that don't exist, quoting quotes that have no support in the law and purport to quote cases that don't exist is well below the standard we expect from the bar.

I can just tell you, unless, you know, again, maybe there's an explanation and maybe you'll have one for us and at some point, and I hope that there is one. I really do. Because I, it is striking and concerning, disappointing, and saddening to think that members of the bar would forward cases to a court that don't exist, and to think that the lawyers on the other side of that didn't read it for whatever reason, didn't check it. It just seems to fall well below the standard that we've all become accustomed to from members of the bar in the second department. Have a nice day everybody.

Michael Sanders: I once again want to apologize to the court and...

Hector Lasalle: You'll have an opportunity to apologize in a different way. Why don't you do your research and find out how that happened though? Okay? We are going to consider, just so you know, we're going to consider whether or not an order to show cause for sanction should be handled. You'll have an, and if we do decide to have one of, if we decide to issue one, uh, you'll have an opportunity to explain why we shouldn't do that. Okay? Have a nice day.

John: Okay. So here's what I take from that. First off, nobody mentioned AI, and as I'll get to in a moment, I actually think that's kind of the whole point. But note how incredulous the judges were that nobody caught the errors. Obviously, and rightfully, they focused the bulk of their ire on the lawyer who filed the problematic brief. But they are genuinely flabbergasted that neither of the opposing counsel had even the barest whiff that anything was wrong.

Why not? As they both said, they were focused on their own arguments, not the details of their opponents. So are they all just terrible lawyers? I'm not sure. It's possible. But I've said before, I avoid attributing to incompetence that which can be adequately explained by overwhelm. I've seen too many of these cases lately to believe that everyone who makes a bad filing is a bad lawyer.

So let me give you two main takeaways. First, this sort of problem is the foreseeable result of an overloaded law practice. When lawyers and their teams get overwhelmed, they turn to shortcuts and shortcuts breed quality problems. Like I said, we don't know for sure that the particular shortcut here was AI related, but it doesn't really matter. Shortcuts lead to unpredictable quality. Full stop.

Second, these problems are also the foreseeable result of poorly defined quality assurance practices. And I think there's a difference between quality review and quality assurance. Quality review is something an individual does. It's subject to all of the variability and subjectivity that comes with being human. Quality assurance is something a system guarantees, explicit standards and checkpoints that hold regardless of who the reviewer is.

I often complain that the typical lawyer's version of a quality assurance standard is, "I know it when I see it." But the problem is that AI produces work that looks right even when it isn't. And that's a dangerous combination. I think the answer is to stop relying on individual vigilance and build quality assurance into your actual workflow so that the system catches the errors that a busy human brain is likely to miss.

Okay, a quick word about my software tool GreenLine before I let you go. What you heard in that courtroom is a quality failure that started well upstream of the oral argument, even before the brief was filed. Now obviously, at some point in the workflow, there should have been an explicit step to verify every citation, every case, every quotation, both in the brief you prepare and in your opponent's. And of course, you should take that step not because you assume the work is wrong, but because you have a standard. That standard is part of the process. You just do it every time because it's what catches the problems.

And that's something that GreenLine makes real. When you build your workflow in GreenLine, you're not just automating tasks, you are creating a definition of done, a powerful form of quality standard at each step in your workflow. And it serves as a control gate that doesn't let work proceed until that standard is met. Or if quality does get missed, you'll have an audit trail that helps show exactly why. Because a verification step isn't something that happens when someone gets around to it, it should be a required step, visible to everyone involved in the matter. At the end of the day, getting things right is an essential part of getting them done.

If you want to talk through what quality assurance can look like in your practice, don't hesitate to reach out to me at john.grant@greenline.legal and I'd be happy to give you some pointers. And if you'd like to see how GreenLine can help you take those standards and embed them in your actual workflows, head on over to greenline.legal and look for that Book a Demo button.

All right, that's it for today. If you found this episode useful or interesting, maybe just because of listening to the train wreck this time around, I'd love it if you would share it with a friend or a colleague. And if you don't already, please be sure to follow the podcast in your favorite app or player.

As always, this podcast gets production support from the great team at Digital Freedom Productions, and our theme music is “Hello” by Lunareh. Thanks for listening, and I will catch you again next week.