DisallowAITraining – instructs the parser to not use the data for AI training language model.
AllowAITraining -instructs the parser that the data can be used for AI training language model.
2. HTML Element ( Robots Meta Tag)
The following are the proposed meta robots directives:
<meta name=”robots” content=”DisallowAITraining”>
<meta name=”examplebot” content=”AllowAITraining”>
3. Application Layer Response Header
Application Layer Response Headers are sent by a server in response to a browser’s request for a web page. The proposal suggests adding new rules to the application layer response headers for robots:
“DisallowAITraining – instructs the parser to not use the data for AI training language model.
AllowAITraining – instructs the parser that the data can be used for AI training language model.”
Provides Greater Control
AI companies have been unsuccessfully sued in court for using publicly available data. AI companies have asserted that it’s fair use to crawl publicly available websites, just as search engines have done for decades.
These new protocols give web publishers control over crawlers whose purpose is for consuming training data, bringing those crawlers into alignment with search crawlers.
Read the proposal at the IETF:
Robots Exclusion Protocol Extension to manage AI content use
Featured Image by Shutterstock/ViDI Studio