The New York City Department of Consumer and Worker Protection (“DCWP”) recently issued a Notice of Adoption of Final Rule (“Final Rule”) relating to the implementation of New York City’s law regulating the use of automated employment decision tools (“AEDT”) by NYC employers and employment agencies.
NYC’s Local Law 144 now takes effect on July 5, 2023. As discussed in our prior post, Local Law 144 prohibits employers and employment agencies from using certain Artificial Intelligence (“AI”) tools in the hiring or promotion process unless the tool has been subject to a bias audit within one year prior to its use, the results of the audit are publicly available, and notice requirements to employees or job candidates are satisfied.
The issuance of DCWP’s Final Rule follows the prior release of two sets of proposed rules in September 2022 and December 2022. The Final Rule’s most significant updates from the December 2022 proposal include an expansion of the definition of AEDTs and modifications to the requirements for bias audits. Key provisions of the Final Rule are summarized below.
What is an Automated Employment Decision Tool?
Local Law 144 defines AEDTs as “any computational process, derived from machine learning, statistical modeling, data analytics, or artificial intelligence, that issues simplified output, including a score, classification, or recommendation, that is used to substantially assist or replace discretionary decision making for making employment decisions that impact natural persons.”
Under the Final Rule, “machine learning, statistical modeling, data analytics, or artificial intelligence” is defined as a group of mathematical, computer-based techniques that:
- generate a prediction, meaning an expected outcome for an observation, such as an assessment of a candidate’s fit or likelihood of success, or that generate a classification, meaning an assignment of an observation to a group, such as categorizations based on skill sets or aptitude; and
- for which a computer at least in part identifies the inputs, the relative importance placed on those inputs, and, if applicable, other parameters for the models in order to improve the accuracy of the prediction or classification.
The Final Rule clarifies which tools fall within the scope of the law by defining the phrase “to substantially assist or replace discretionary decision making” as:
- relying “solely on a simplified output (score, tag, classification, ranking, etc.) with no other factors considered”;
- using the tool’s “output as one of a set of criteria where the simplified output is weighted more than any other criterion in the set”; or
- using the tool’s “output to overrule conclusions derived from other factors including human decision-making.”
Employers and employment agencies should be aware of four categories of requirements related to bias audits:
- (i) Structure and Required Calculations;
- (ii) Permissible Data;
- (iii) Independent Auditor; and
- (iv) Publication of Results.
Each category is addressed in more detail below.
Structure and Required Calculations
The Final Rule details the structure and requirements for bias audits, with new requirements following prior releases of proposed rules. An AEDT cannot be used if more than one year has passed since the most recent bias audit. Bias audits must adhere to the following:
- Where an AEDT selects individuals to move forward in the hiring process or classifies individuals into groups, the bias audit must:
- (i) calculate the selection rate for each category;
- (ii) calculate the impact ratio for each category; and
- (iii) indicate the number of individuals the AEDT assessed who are not included because they fall within an unknown category (e.g., applicants who declined to disclose demographic data).
Categories mirror the EEO-protected categories reported on the U.S. Equal Employment Opportunity Commission’s EEO-1 Component 1 report. These categories include race, ethnicity, and sex.
- Where the AEDT only scores individuals rather than selecting them, the bias audit must:
- (i) calculate the median score for the full sample of applicants;
- (ii) calculate the rate at which individuals receive a score above the sample’s median score in the each category/classification;
- (iii) calculate the impact ratio for each category/classification; and
- (iv) indicate the number of individuals the AEDT assessed who are not included because they fall within an unknown category.
The impact ratio must be calculated either as (i) a selection rate for a category divided by the selection rate of the most selected category; or (ii) a scoring rate for a category divided by the scoring rate of the highest scoring category. Impact ratio calculations may exclude a category that makes up less than 2% of the data being used for the bias audit.
The Final Rule also indicates that the required calculations described above must be conducted for standalone sex, race, and ethnicity categories (e.g., Male, Female, Hispanic or Latino, Black, Asian, White, etc.), as well as intersectional groupings (e.g., Black Females, White Males, etc.).
Permissible Data
Bias audits must use “historical data,” which is defined as “data collected during an employer or an employment agency’s use of an AEDT to assess candidates for employment or employees for promotion.” Under the Final Rule, a bias audit may rely on historical data of other employers or employment agencies, but only if the employer or employment agency (i) has provided the independent auditor with historical data from its own use of the AEDT or (ii) has never used the AEDT.
Alternatively, test data may be used if there is insufficient historical data available for a statistically significant bias audit. If test data is used, a summary of results of the bias audit must explain why historical data was not used, as well as describe how the test data was generated and obtained.
Independent Auditor
An “independent auditor” must perform bias audits. The Final Rule clarifies that an “independent auditor” is “a person or group that is capable of exercising objective and impartial judgment on all issues within the scope of a bias audit of an AEDT.” An auditor is not independent if the auditor:
- (i) is or was involved in using, developing, or distributing the AEDT;
- (ii) has an employment relationship with an employer or employment agency that uses AEDT; or
- (iii) has a direct or material indirect financial interest in an employer or employment agency that uses the AEDT.
Similarly, an auditor is not independent if it has an employment relationship with or financial interest in a vendor that developed or distributes the AEDT.
Publication of Results
Local Law 144 requires that the results of a bias audit must be “made publicly available on the website of the employer or employment agency.” The Final Rule clarifies that the published results — the date of the most recent bias audit, summary of results, and distribution date of the AEDT — must be posted on the employment section of the entity’s website in a “clear and conspicuous manner.”
The summary of results must include the source and an explanation of the data used to conduct the audit; the number of individuals who fall within an unknown category; and the number of individuals, selection or scoring rates, and impact ratios for all categories. If a category comprising less than 2% of the data being used for the bias audit is excluded from the required calculations for impact ratios, the summary must include the independent auditor’s justification, as well as the number of applicants and scoring rate or selection rate for the excluded category.
The published results must remain posted for at least six months after the AEDT was last used to make an employment decision.
Any changes to the notice requirements?
Local Law 144 requires that any employer or employment agency that uses an AEDT to screen an employee or a candidate who has applied for a position for an employment decision must notify individuals who reside in New York City that the AEDT will be used in connection with their assessment or evaluation, as well as the job qualifications and characteristics that the AEDT will consider. Notice must be provided at least 10 business days before use of an AEDT and, notably, must include instructions for how to request an alternative selection process or accommodation.
The Final Rule’s notice provisions remain similar to those included in the proposed rules in September 2022 and December 2022. Importantly, the Final Rule clarifies that Local Law 144 only requires employers or employment agencies to include how an individual might request alternative selection processes or accommodation to the extent such options are “available.” While employers or employment agencies must still comply with reasonable accommodation obligations under other laws, the Final Rule states that “[n]othing under [Local Law 144] requires [them] to provide an alternative selection process.”
Looking Ahead
As an immediate next step, employers and employment agencies should identify whether their hiring and promotion efforts leverage AI tools that fall within the scope of NYC Local Law 144 and adjust their processes accordingly in advance of the July 5, 2023 enforcement date.
While NYC’s Local Law 144 is groundbreaking, it is likely only a small part of what will become an increasingly complex regulatory environment related to AI and machine learning. Companies should prepare to comply with more laws and regulations as federal, state, and local legislatures take stock of the use of AI in various decision-making processes. Covington will continue to monitor developments and publish relevant updates. In the interim, if you have any questions about the material covered above, please contact Covington members of our Employment, Data Privacy, and Technology groups.