In the summer of 2017, three Wisconsin teenagers were killed in a high-speed car crash. At the time of the collision, the boys were recording their speed using Snapchat's Speed Filter—123 miles per hour. This was not the first such incident: The same filter was linked to several other crashes between 2015 and 2017.
Parents of the Wisconsin teenagers sued Snapchat, claiming that its product, which awarded “trophies, streaks, and social recognition” to users who topped 100 miles per hour, was negligently designed to encourage dangerous high-speed driving. A lower court initially found that Section 230 of the Communications Decency Act immunized Snapchat from responsibility, claiming the app wasn’t liable for third-party content created by people using its Speed Filter. But in 2021 the Ninth Circuit reversed the lower court's ruling.
Platforms are largely immune from being held liable for this kind of content due to Section 230. But, in this important case–Lemmon v. Snap–the Ninth Circuit made a critical distinction between a platform’s own harmful product design and its hosting of harmful third-party content. The argument wasn’t that Snapchat had created or hosted harmful content, but rather that it had negligently designed a feature, the Speed Filter, that incentivized dangerous behavior. The Ninth Circuit correctly found that the lower court erred in invoking Section 230 as a defense. It was the wrong legal instrument. Instead, the court turned its focus to Snapchat’s negligent design of the Speed Filter—a common product liability tort.
Frustratingly, in the intervening years, and most recently in last month’s US Supreme Court oral arguments for Gonzalez v. Google, the courts have failed to understand or distinguish between harmful content and harmful design choices. Judges hearing these cases, and legislators working to rein in online abuses and harmful activity, must keep this distinction in mind and focus on platforms’ negligent product design rather than becoming distracted by broad claims of Section 230 immunity over harmful content.
At the heart of Gonzalez is the question of whether Section 230 protects YouTube not only when it hosts third-party content, but also when it makes targeted recommendations for what users should watch. Gonzalez’s attorney argued that YouTube should not receive Section 230 immunity for recommending videos, claiming that the act of curating and recommending what third-party material it displays is content creation in its own right. Google’s attorney retorted that its recommendation algorithm is neutral, treating all content it recommends to users in the same way. But these arguments miss the mark. There’s no need to invoke Section 230 at all in order to prevent the harms being considered in this case. It’s not that YouTube’s recommendation feature created new content, but that the “neutral” recommendation algorithms are negligently designed to not differentiate between, say, ISIS videos and cat videos. In fact, recommendations actively favor harmful and dangerous content.
Recommendation features like YouTube's Watch Next and Recommended for You–which lie at the core of Gonzalez–materially contribute to harm because they prioritize outrageous and sensational material, and they encourage and monetarily reward users for creating such content. YouTube designed its recommendation features to increase user engagement and ad revenue. The creators of this system should have known that it would encourage and promote harmful behavior.