Governing machine-learning models: challenging the personal data presumption

International Data Privacy Law, vol. 10, pp. 187–200

Abstract

This article confronts assertions made by Dr Michael Veale, Dr Reuben Binns, and Professor Lilian Edwards in ‘Algorithms that remember: Model Inversion Attacks and Data Protection Law’, as well as the general trend by the courts to broaden the definition of ‘personal data’ under Article 4(1) GDPR to include ‘everything data-related’. Veale and others use examples from computer science to suggest some models, subject to certain attacks, reveal personal data. Accordingly, Veale and others argue that data subject rights could be exercised against the model itself. A computer science perspective, as well as case law from the Court of Justice of the European Union, is used to argue that effective machine-learning model governance can be achieved without widening the scope of personal data and that the governance of machine-learning models is better achieved through already existing provisions of data protection and other areas of law. Extending the scope of personal data to machine-learning models would render the protections granted to intelligent endeavours within the black box ineffectual. There are growing calls for regulation of models used to make inferences about an individual. For example, in its final report on ‘Disinformation and fake news’, the UK’s Department of Culture, Media and Sport specifically called for the extension of protections of privacy law ‘beyond personal information to include models used to make inferences about an individual’.1 Arguably, this reflects the general trend of the CJEU to extend the meaning of ‘personal data’ to almost everything data-related.2 Widening the definition might be aimed at the benefit of data subjects in an exponentially growing, technological-connected world and imposes new obligations on controllers and processors in multiple ‘smart’ environments, but it imposes regulatory burdens on holders of information and may also reduce the effectivity of data subject tools for ensuring fundamental rights are protected.3 In ‘Algorithms that remember: Model inversion attacks and data protection law’,4 legal and computer science scholars Veale, Edwards, and Binns outline how rapidly changing technology could theoretically render vulnerable machine-learning models,5 thought normally to be protected by obligations of confidence, as personal data.6 The authors of ‘Algorithms that remember’ offer a cautionary tale that models in machine-learning systems can be classified as personal data under European data protection law and that the General Data Protection Regulation (GDPR) could be seen as an important tool in the governance of decision-making models. As a consequence of classifying models as personal data, certain data subject rights (eg rights of access) and obligations on data controllers (eg to provide data subjects meaningful information about logic in decision-making/erasure/rectification) are activated. Certain type of ‘attacks’ on models ‘may leak data they were trained with’; as a consequence, ‘data protection rights and obligations might then apply to models themselves’.7 In the narrowest sense, the authors reflect upon research from computer science scholarship that highlights how machine-learning models can be subject to a ‘range of cybersecurity attacks that causes breaches of confidentiality’ and that some of these attacks reveal personal data.8 Although the authors do not expressly state as such, we have proceeded on the basis that Veale and others’s first claim, under certain circumstances models can be classified as personal data, is normative; and that their second argument: ‘cybersecurity attacks on machine learning models can reveal personal data; therefore, models are subject to data protection obligations’ is descriptive.9 This amalgamation of both and through their of the is in this the made by Veale and others from a computer science to and model inversion attacks reveal an in the by Article 4(1) and the of the Court of Justice of the European As to inversion and models can data. an might data, this not make the model Veale and others their about models may personal data by a of to the models an but on a vulnerable model would the rights and obligations of a data under European data protection we the legal of an on a model from both a computer science and legal that the attacks are not within the data protection law is and the other legal are better to better models. from the CJEU that as it is not to the data the legal for data subject rights is not a of the Veale and we by a of how models the then the case law from the of the CJEU and from computer science to the of attacks on machine-learning models. Veale and others that have by their to models the protection as personal data of data subjects from data. argue that this is not the then their claim, from a of computer science personal data and models are personal data. the case law of the CJEU to the of model inversion and is legal basis at in the GDPR the case law of the CJEU for of and inferences how to upon that black are personal and data, and can without to legal therefore, be seen as that are subject to a of regulatory This that legal the black data protection apply to personal data, machine-learning models be as Extending personal data protection to models would rights and and the black we argue that data protection law have a machine-learning in ensuring rights personal data in the data and ensuring the that areas of the law of the GDPR are better to the protection of data this without the scope of personal data to machine-learning models. For the of this we have Veale and as machine-learning systems data a model that can provide of new data on the basis of from data. a model not the data a of in computer science that in certain to reveal information about the data in the In of the data is in it personal data, the of is to as an in as model inversion attacks and The authors suggest in attacks can be used to information about model that for be as personal and others that the GDPR could an important tool for the regulation and governance of machine-learning models. and Veale and others suggest that the in and and a of on the and scope of and ‘personal data’ from Article both the regulatory and to the scope of personal As the scope of personal data law by the and data protection its meaning to a of information to an is not in the of In they a of data protection scholars that that the CJEU is widening the scope of personal data to include data that be to be the of the of Veale and others the we this the scope of personal data, and the for it as the law not in this do not to the by but to that the made models might under the definition of personal not on the of definition that to The authors the of widening the scope of personal data law and to the of classifying models as personal data. they the definition were to be in scope the scope of personal data as and others and the models would In model inversion and attacks, do models as personal data without to a of data protection law. This is by the of the to to and the of its scope and of the and of the GDPR are as for data subjects to meaningful information about an that of the data a is ‘personal that to a The GDPR ‘personal data’ as information to an an an is can be in by to an as a an data, an to to the of that data are used to the models that the of the and the is to personal data, the data also personal data. Veale and others their of models as personal data on the basis that data from model inversion attacks are as personal data, as they a can be used to that this not that machine-learning models be classified as personal models that are vulnerable to attacks the model inversion attacks, as Veale and others in their In the we that models be as personal data. are in not data information about can be the of the data are of within data, also data for example, and for the a in the data other information is not is also the attacks are on the basis of the as a black argue in the that the information in the models is data, from it they not be classified as personal data. The GDPR a to the at the a data personal data for the of decision-making and the This rights to information about the logic as well as and the of for the data The information of to the data subject about the the on in the without a of the used of the In to the that some machine-learning models be classified as personal data, it is to on how models are and how they a machine-learning model is a that at classifying as This is by machine-learning the and the in the data and the The learning is about a can be as a on new data to a to a The model then a of the of the on the for the it from the data. models have an important in In systems the would be on a model a that a on on In this the of the black box to by Veale and others are not data. and have used to data, but the information the model is In systems on machine this is by the machine-learning model is a of without for is important to that the of the machine-learning the is of a it to a and that to be effective and an of that as we may of the about the decision-making can be to is by In a of in the machine-learning we to how we to to an of the of a machine this be as an a of the of a an (eg of is also important not to a machine-learning model a data do not in the the in the data and in the model may be to of certain data this not that personal data can be within the how the by the on the data, and how of the may be and others trained a model to of from of the model it could be that in to from without that the data that were in the that of the of the in the is we the to we this the made by the and the from the may its on of the that are not to is this that the upon we then and that the meaning of logic is not to a of the as such, but to the logic we on that that logic may be (eg the trained on This is Article is not to logic it is not the model but of it to a logic that is not in the the In their on how models can reveal personal data, Veale and others the from systems be to the could use the to an from a that it is not the machine-learning model that the is the may a from a on the basis of a that through an of the This in that that this in the data of the This could reveal information about the data subject the use of the model is (eg the trained on a data of of a certain but that information not leak from the but from then the model be as personal of the of models can be of as personal data, by at the of data, and the definition of personal data. The of data and information are as in an is the to is the to is and is the of information to and Data are as that of and their are the of this data and information are as In the of machine data for example, through and that is not a they would as models in that they make the of information the definition of personal data in Article 4(1) of the GDPR as information the for the of this it would be an to models in the of personal data, in the models as data are In that sense, machine-learning models are from they also to information from their personal data were not at in the of the models are on data of a they do not data, personal data. models information on the certain (eg to a on the basis of their in the case of the In the case of models, attacks the of a data in the of the is to information about a models are not to be personal it from Veale and of in the of attacks that they be to be personal data as have a to it is personal and subject to the GDPR and not subject to the data protection This to data protection and computer science scholars about the of can be In this data data that do not an from the data from that data in other data, of the that are to be used to data are are then data are not by data protection The of data protection not apply to information that not to an to personal data in a that the data subject is not The scope of the Article data controllers to personal in to a data that is not by the of obligations to personal are as in data A report from the as as to that data be data is to an some technology to this effective to the through and a on the report that to be to on their argue that the used not at on the they how an data are about an that to of the in the data were to of they not Veale and others suggest a can be made to personal data have This is on that the to the personal data used in the is that data personal data. a data in of both the model and the to personal data be the in law as a data that a model the is by a The authors that imposes a of of but that model inversion and attacks would render the in as as the to the data is by The on personal data from the data through model inversion to about the of and data. to attacks on data the model is not personal data, as an of the of the model in to information from the data that can be to a the model data, as personal data, is a the model not the information to a the of of data, is the tool in the The information to the the it to the is on a by a that on a of for example, about the of the is important to is not as such, in the an model as personal data an that in an by a a CJEU on the of ‘personal it to on the basis of machine-learning are As by the the of the data may the is for the attacks by Veale and others is that models they on data that were in the data they were trained to they on data. Model attacks are about a certain data about a as a this personal data in the data personal data were in the data by information on the model is trained reveal of about that (eg the model is aimed at for a certain the of type of make a machine-learning model personal data, and under the regulatory of the ‘attacks’ are not in a sense, but in the that to can be in the of The use of in inversion not to some protection (eg for the to a the model an to a of models that the The is a that in the to the then the that trained models data from the data data for the first are in machine machine-learning make their in a that data in the and can also be from The a data of the data their their the model is personal data, but of the they the personal data in the they This the to provide for this and the of the data controllers In the systems the to a is not from the data and the data are not data that the can to data. There is for the data to a data. The may be subject to attacks that could a of in some to an some of this on a model that it personal data. a model in personal data, Veale and others suggest that data subjects their Article GDPR to of suggest to the model on an data the personal data of the data The model this data subject to be as the its they suggest that the model be to the of the data in the of the the for of the this is a but at not a for that are not both do not of the but of data within the logic of the Veale and others models as personal data in the can be personal data for multiple data data subject for of an to of the rights of the other data models not be a but not models be to as personal data. not be classified as personal data in to to provide protection from the of is within both the model the data This would be a data subjects the to that a model is on data. to the case law of the the GDPR to data are to an As the this is for about data subjects can their rights against model For the some we that be to include of its it is important to that the not the Article on the meaning of and to provide on the The consequence of this is to the data protection of the of data protection some argue that include at the can include information that could information that could be personal data, that the other of Article 4(1) GDPR are The in meaning to the but that scope is in can about from data by data controllers by legal data could personal data data in the of the to the legal an and an for an of information that would render data The that the could to information from the in to against of it not to as personal data a of it may of to The by Veale and others in apply to the of the of information a to be to the data The that the of of information to the of is would be on of the that it would a in of and then of information would not personal a by an a a that the to the personal data within the meaning of that in to that the the legal it to the data subject data the about that The to a a data an that be to and an that can the of the the The that an could personal data for the the the legal to information that the to the of the legal the and the The information that could be the to the model inversion and attacks are not the legal by the the of the data subject by law on of the that it a in of and that the of in to be This that to be to in a to a data subject is be of the to be as by the by to the are to be used to the be of as the of and the of for the technology at the of the and this not include by law. The is that attacks reveal that could personal data. The model may not have the legal to from information that the model to the The examples by Veale and others are to the model and the are by the data and the and the model are by is not by Veale and others is the model is to reveal information that by in a that is would also be an for that data to the obligations of data protection law to data subjects without protections an is used to personal data. also an to the obligations on data that of an The would be obligations from personal data and inferences about the of data Accordingly, the would be against the have made a data without a legal basis for the of personal data. This would to the for for Article in be for the by this Article in by Veale and be from under it that it is not in for the to the data subjects could their rights against a they have not to their but is a a to their it is that use a that the data to their and the of Article is data subjects would not be to as a of an this data subjects be that a data subject to their rights be to a data to personal data to and upon the Veale and The data that an might personal data a for data protection the new data, the data to the data There would be data within a machine-learning model that to an individual. The data subject would be to the of information against the of the data and against the of both and model can be seen in the law in the UK’s Data Protection The attacks that Veale and others are under and it an for a to without the of the to without the to it without the of the a model is made of data, then it is an to information that is personal data. The is that model inversion and attacks are is a to a model for the of personal data to data. from the for have used to and are fundamental to a models are vulnerable to and other of cybersecurity and systems are protected through legal against both and The as a of of to a of The also to in their law a for computer without The of the is against the and of computer and computer data as well as the of and data by for the of data in a data without would to for and could be under and of the UK’s In made that to of a computer is not from for other of data of the as a to data to other data from an model attacks could also to of computer a is to be used in a certain this not that to reveal information about a data subject is not an to data not extend to to that the an information that is personal data without the of the for the personal data’ specifically as a to these of in the it an for a to personal data that are information that the without the of the for the personal data, and in circumstances in the an under these provisions be important tools to the of models. and data protection the also the for ensuring models that have through their are and data’ a of for to from the data and of to the and of data use and data of the for models as some an benefit in the of data, some are from argue that the machine-learning the black box is is of the for machine-learning models have to other This is machine-learning models in that the the model the and the model to the also apply to is that have As models they can both and in the is of the to the The for for its to provide protection for data subjects the in for a in without a on that The in machine-learning from the of data the of a certain of for the models that are of to for certain are to to to the models. the other the GDPR on ensuring data controllers to data the use and that on how personal data are For example, the GDPR to the of of and The is by and not by the This the of the that an to their under Article of the to the that a data and under Article As a of it to that the the to use of the information within a the it be that the information is The GDPR the of personal data, and to the and of personal data. the other the and to and of The that include everything from to and other and of are in the GDPR the on The not the to data of the GDPR an to an and for ensuring the rights and of data the and the of the be by to the and of the and of the to the rights and of the data subject be by to the and of the be on the basis of an by it is data a a Article also the state of the the of and the and of as well as the of and for rights and of by the the both at the of the of the for and at the of the and a on would not of the Article an of the of an the other in to and the of a machine-learning model protected by law to they other the scope of and and provisions and effective the use of a of the and and a on the of a model the of the they their obligations in the GDPR to and to personal data’ as as they are can be from the This is in the are to the of models that the of protection as a have also the legal for under the GDPR and be seen as a for the protection and governance of machine-learning models. The GDPR to be a tool to and data protection for within the European and important in data privacy regulation in have that the GDPR data subjects new rights and data protection new to the the and classifying models as personal data the personal and data in a the GDPR and the scope of personal data at would a and it would be for data subjects to their rights for data protection to do meaningful The personal and data and but also and new and Model is a consequence of the to the of information data and to of data for In some the model trained to that it (eg and and to the that it protection as to that systems are of of the on decision-making to in about and have to that widening the scope of personal data to include is the to fundamental rights the and in we For example, Edwards and Veale have that to these is to the scope of the GDPR to include and of the to widening the scope of personal data to include data that have a to can be from the for against be in widening the scope of personal data, the of the GDPR rights and obligations There are other that apply and to the fundamental rights at Data protection is not to be the fundamental that can be to the GDPR already obligations for data controllers to of systems and to and classifying of these systems as personal data the of use it the data subject is a better Veale and others models that under the definition of personal data on the basis of it to information on the of the their data were used in the data it is important to be on the that the model not personal data reveal of a to reveal the personal data, for example, the of the black box model a model for the of information from the data. This as use of the do not have to for a in of data protection we already have a of of the we the to the of the model are to an Veale and others model for a that of in the data of the is not by the model but by a The data in personal data on the basis of data can not data that are personal data. this is not the of a trend in law in privacy and data use of to the scope of ‘personal to the that the of information technology to the is in this that make in computer systems existing within data protection law. Data protection law is not to from attacks, but already The from existing legal the scope of the of data, and it of we the of their to against of data systems protection we can provide for ‘data the and of the GDPR as the would be to that information to be ‘personal then would be for to have the that data in the of a to data subjects not make a of a model a data are in to the protection for the and the to the data subjects as existing in the data, the model would not to personal data. machine-learning systems their the that be as as it is The of machine-learning systems to the existing and information The of Veale and others’s would in the protection of models as personal data as as and they have This is the of This might is not of in not models as personal data. without would it be to of a Veale and others that personal data a model is an of the to this is not of not in a is in to The is an of the data protection not others to the that For the we the to be the they to

Authors 2

1. Mark R. Leiser corresponding Aachen M. R. Leiser Leiden Law School/eLaw Francien Dechesne

   Leiden University

   Affiliation as printed

   M. R. Leiser, Leiden Law School/eLaw, Leiden University, Leiden, The Netherlands

   Francien Dechesne, Leiden Law School/eLaw, Leiden University, Leiden, The Netherlands

2. Francien Dechesne Aachen Leiden Law School/eLaw

   Leiden University

   Affiliation as printed

   Leiden Law School/eLaw, Leiden University , Leiden, The Netherlands. Email: f.dechesne@law.leidenuniv.nl