In recent years the availability of huge databases and the increased capacity of computers have transformed many fields in science and technology as the neural networks, renowned as procedures for “ Deep learning ” to avoid the negative connotations of its failure in the years 60 and 70 , or genetic. Economic research has also been influenced by the availability of databases of increasing size and granularity. this way the empirical research, as in the years 70 it was marginal ( 11 per cent of the most cited in academic journals) against the dominance of the theory ( 77 %), currently reaches the 60 % 10 per cent of the theoretical articles ¹ .

In this work presents a vision of this transition towards the use of large databases in the field of labour economics in the case of spain ² The course arises from a personal perspective from lessons learned from my own career as an investigator. Many of the thoughts set out below matured from reflections submitted García Montalvo ( 2021 a, b, c) and García Montalvo ( 2014 ). Although my first work using massive data back to the 1995 , and this relates to the negotiation in real time in futures market bonds, in this article i will focus exclusively on my experiences related to research in the labour market ³ .

The history of the pervasiveness of large databases ⁴ work in spain is going through several stages. The first is that it would place prehistory, characterized by the principle of “ guisas you what you, what comes ” bikepark. The second phase would be the Average age of labour statistics characterized by the construction of the grand cathedral, the sample Of Working Lives (MCVL), and the beginning of an opening of the producers of statistics to the demands of researchers. The third stage or Modern Age, which we could also denominadar Israeliano such as the period, is characterized by more generalised access, but under the principle of “ tell me who know and i will tell you if you get ” and “ do it before the change of responsible ”. In the Contemporary Age goes from pervasive mistrust limited collaboration and from here there are two borders: the first border mark the availability of administrative data merged for almost the totality of the citizens of a country like Denmark and Norway. The case of Iceland even includes a database of the dna of all the country's inhabitants. The second boundary passes through the integration of data from private companies and administrative data. This is the current border line in the construction of large databases for economic research.

In the mid- 90 there was a widespread view from the spanish university system as a “ factory unemployed ”, which was based on the high unemployment rate of university graduates spaniards. Indeed, the active population survey showed that the children of young university 29 years suffer higher unemployment rates to graduates of Vocational Training and similar to the graduates of basic general education or That. Only the group of young people without studies were higher rates. Unfortunately the EPA was insufficiently granularity to assess whether there were differences by qualifications, or which are factors that determined the success or failure in the integration of the young university students in the labour market. In those days the data of Social security were not an option and, therefore, in an attempt to analyse the underlying determinants of young people into work, we contacted many universities to solicit their collaborative programming and development of a specific survey. The talks with universities failed to fear that might occur, subsequently, comparisons between universities.

To this situation, with the financing of the european commission (programme TSER), and in collaboration with research centres of other 10 european countries (Austria, czech republic, finland, france, germany, italy, norway, sweden, united kingdom and netherlands) and japan, we develop the project CHEERS (Carreer after Higher Education: a European Research Survey). For more than a year in the discussion on the construction of a comprehensive questionnaire, to national adjustments, which allowed a follow-up to the early years of university graduates after completion of the studies. Also reported information on access to university studies, the characteristics of the race, competencies and skills developed during the same, as well as the work done since the completion of studies at the time of the survey (four years after completion of the studies). The results of the study set out in a book (Teichler, 2007 ) containing cross-cutting chapters as well as in each country ⁵ . With this information, we learnt that the spanish university system was not a factory of unemployed, but that was a factory of sobrecualificados, at least subjective ⁶ .

The questionnaire drafted by project participants had enough subsequently impact because many operations subsequent statistics have taken it as a basis, including the time frame ad-hoc of four years after graduation as a determinant of target population. The first adoption took place in the survey into work of the catalan University HERE, now 8 editions ( 2001 - 2005 - 2008 - 2011 - 2014 - 2017 - 2020 - 2023 ). The INE has also developed two surveys on insertion of university graduates (EILU), one in 2014 (referring to graduates of 2009 - 10 ) and the second in 2019 (reference to local graduates 2013 - 14 ). At present Social security also publishes a report on the employment of young university graduates four years after the end of their studies.

The average age of labour statistics in Spain begins with the construction of the “ Grand Cathedral ” that was the sample of Working Lives ( 2004 ) which marks a huge change compared to the requests ad-hoc of prior periods or the use of statistics “ prêt-à-porter ”. From 2006 integrated fiscal data. The MCVL also integrates the Continuous Roll with what is an essential source of information on the spanish labour market. The widespread availability of MCVL provoked an explosion in the investigation into the labour market and the pension system in Spain.

The Modern Age in the economic research with large databases begins with a number of proposals. In America Card et al. ( 2010 ) propose to expand access to administrative data for economic research at what they described as “ erosion of the us advantage in the creation of data for economic research ”. Card et al. ( 2010 ) the importance for the social sciences of high-quality data and underline that administrative data are much higher than traditional sample surveys: larger size (total population), structure, minor longitudinal measurement error, imputation and non-participation, etc. The priority for research is to develop a director and secure access to administrative data for research. Card et al. ( 2010 ) propose a mechanism, similar to the scandinavian countries, to develop the system whereby the national statistical agency gets administrative information on all other generators of statistics, prepares the anonimización of data for each research project, provides a secure access to researchers and eventually, only the outcome of the inquiry is public. In this process it is noted that direct access to microdata is critical to the Success of the programme. The synthetic data or the sending of programmes to agencies from outside are not considered good alternatives.

In the united kingdom 2014 launching this Administrative Data Research Network (ADRN) as a collaboration between universities, government agencies, statistical agencies and researchers to facilitate access to administrative data merged among various sources. The initiative had borne fruit as huge comment afterwards.

The Final milestone in this series of international initiatives is the Opportunity Insight Lab founded by Raj Chetty, John Friedman and Nathaniel Hendren in 2018 with the financing of Chan Zuckerberg Initiative. A few years before they had access to data of the Tax authority of United States for economic research, not without some dispute about the objectives of the exploration and use of data. The information provided by the Agencia Tributaria. allowed research very interesting and impossible to earlier, including the famous Atlas opportunities which showed, among many other things, such as social mobility in the united states had dropped sharply between 1940 and half years 80 .

Meanwhile, in Spain, the Agencia Tributaria also is the agency with the databases more comprehensive and “ learning ” machine. Surely there is no other tax bureau at the world with the ability of the spanish agency. The AEAT possesses the database Zujar fed by 30 billions of data, and uses a variety of tools for analysis (Theseus, Electra, Midas – ROI, Genio-Infonor, procedures of “ site scraping ”, etc.).

However, in the mid- 2010 still, it is difficult to work with administrative data of spanish institutions. In addition, in case of the right person and that this is the provision of help, for each investigation legal and technical infrastructure of some complexity. 2015 i had the opportunity to work with some data on random assignment of subsidized housing in the País Vasco . The aim was to analyse the impact of the possession of houses and, in particular, of a debt in the form of a mortgage, on the labour supply. Of Course, information on changes in the labour market, associated with the date of purchase of housing, were not included in the administrative information on the mechanism of allocation of housing. this way had to be signed four conventions, one of them with the treasury of Social security, to merge the information in the form set by the “ Notes file sharing, security and anonimización ” of the treasury. This procedure was an triangularización information so that the investigator has never had access to data identifiers of individuals and only the public administrations had such information for the merger.

My experience during those years with regard to the facilities to obtain data and manage the merging of several databases was mixed. The institutions more sensitive to the needs of investigation were Social security, registrars of property and some self-governments. Among the agencies little sensitive were the General directorate of Trafficking, Tax Bureau and ANCERT (Notarial Statistics) ⁷ .

At the end of year 2010 the AIREF promotes a series of meetings with researchers with the aim of initiating a strategy of access to administrative data. This momentum was not oblivious to the fact that Israel Arroyo had been incorporated into the AIReF in 2018 after to encourage and facilitate, since the post of deputy director-general for budgets, Economic Studies and statistics in the treasury of Social security, smooth collaboration of the Social security with the researchers on subjects related to the labour market. As a result of these meetings and internal reflection of the published AIReF View 1 / 20 (AIReF, 2020 ), which poses a strategy proposal for access to administrative data in spain, as well as a number of policy changes required to do so.

At the end of the 1990s 2010 the Agencia Tributaria also starts relative openness which culminates with the collaboration with the national statistics institute in the production of the Atlas of the distribution of household income ( 2019 ), which has a huge success, and the facilitation of the data required for Atlas opportunities in Spain ( 2020 ) by unf Felipe gonzález and COTEC but, unfortunately, this second implementation is far less known ⁸ . Tax Authority also participated very actively, and was instrumental in the provision of information, in the construction of the state system of indices of Rental housing, which provides information rental prices up district level census. Another sign of this new time is that the AEAT enable AIReF used anonimizas income ( 35 , 5 millions) that have been crossed with the Family Budget survey to analyse the joint taxation and vat INCOME of families (AIReF, 2020 ), although these data are not public.

In recent years progress has also been made in other administrative data. The PET (panel empresa-trabajadores data) provides a similar vision to MCVL but from the perspective of the company from link of records on business and their working lives of workers. Moreover, the establishment of laboratory of bank data allowed Spain to the Balance sheets.

Within the partnerships between government agencies and research institutions the DataReSS, a collaboration between the institute of statistics Cataluña (IDESCAT) and Barcelona Graduate School of Economics, facilitates access to records that produce some catalan public administrations.

In the early years 2020 there are two other important milestones. First, establishing the office of the Secretary of state of Digitization and Artificial intelligence. The second milestone is institutional communiqué 13 april 2021 where the INE, Tax, Social security and the bank of spain agree to start working together on the development of a system of access to their databases for scientific purposes of The public interest. 12 february 2024 a new institutional communiqué, this time endorsed by the Social security, the SEPE, INE, the bank of spain and tax authority, reported the signing of an agreement to allow access to its databases for scientific research with public interest. The agreement, which resulted in the ES _ DataLab (es-datalab.es), allows other public institutions can also join in the future.

The nordic model has always been the mirror in which many countries have sought to be reflected in obtaining further the use of administrative data for use in research. Both the statistical offices in Iceland, denmark, norway and sweden allow researchers to major administrative databases already merged ⁹ . The access in the case may also be norwegian intermediate by Norwegian Social Sciences Data Service. More recently, countries such as germany (IAB) and france (CASD) have also set up agencies to facilitate access to large databases. But certainly one of the cases of Success is the initiative of the united kingdom. One of the fundamental grounds for this initiative was, similarly to what has happened in the united states, the feeling that the united kingdom was missing the boat of economic research of border by constraints that had the researchers to access large databases In administrative matters. 2014 the Administrative Data Research Network of the united kingdom (ADRN) as a collaboration between universities, government agencies, statistical agencies and researchers to facilitate access to administrative data already merged among various sources. This initiative gave enormous fruits. In this regard creates enviably healthy than a few months after the beginning of the pandemic of COVID 19 researchers of the united kingdom to be able to merge 17 millions of electronic health records and calculate the probability of death by COVID 19 in terms of dozens of risk factors (Opensafely Collaborative et al., 2020 ).

Although public administrations generated huge amounts of administrative data, private companies still produce as much information. In Addition, private companies provide a spatial and temporal graininess and immediacy which can hardly be achieved with administrative data. The advent of the pandemic of COVID 19 and the need for very high frequency to monitor economic developments, and the impact of policies adopted to mitigate the impact of the pandemic, has become more important rapid availability of large databases, encouraging collaboration público-privada. In this case the term refers to public universities, public bodies and research centres working with large databases of private companies, but you can also refer to public databases built based on microdata from the private companies.

An example of this collaboration was the “ economic tracker ” of the Opportunity Insight whose construction was started in 2020 . Chetty et al. ( 2023 ) describes the data used to track the impact of COVID 19 in various sizes. Between private companies that collaborate contributing data are Affinity Solutions (expenditure on credit and debit cards), CoinOut (cash transactions), Womply (using credit cards in small businesses), Paychex (data on employment and payroll 670 . 000 small and medium-sized enterprises), Intuit (provides payroll companies), Earnin (provides payroll progress which allows access to the income of workers who have been inscribed on the web), Kronos (management service that includes workers 30 . 000 companies and 3 . 2 millions of workers), Homebase (tool of small business management) and Zearn (application for learning mathematics used by 925 . 000 students).

The availability of information on bank transactionality (credit and debit cards, TPV points accounts recruitment of financial assets, etc.) has been of fundamental importance in the revolution of the economy in real time. In recent years have increased the partnerships at international level through agreements between universities and plenty of banks to use the banking information. The list includes as many countries: united states, united kingdom, france, japan, Denmark, sweden, iceland, Portugal, Netherlands, China and spain. The cooperating companies include all types of financial entities since Fintechs until traditional banks. Table 1 provides a summary of countries where there are ongoing investigations and private entities that provide support.

The objective of these contributions is to analyse economic theories as permanent income, the impact of the internal exiles, the impact of policies of transfers of revenue from COVID 19 , the evolution of expenditures during the pandemic distribucional or the impact of the epidemic on expenditure. These databases massive financial institutions also allow us to deal with aspects related to the labour market. One example is the partnership between a team of investigators of the department of economics, university Pompeu Fabra and Caixabank Research in real-time analysis of wage inequality in Spain since the beginning of the epidemic of COVID- 19 (Aspach et al. 2021 ) ¹⁰ the effect of ERTE on wage inequality (Aspach et al. 2022 ), or the impact of the minimum wage, and changing the conditions in 2023 on consumption and employment. It is interesting to note that recently a number of national statistical agencies have approached for financial institutions to analyse the possibility of using your data as a source of statistical indicators.

Another example of use of personal data are produced in the case of collaboration between public institutions or agencies and companies. For example, the european Central bank (ECB) used by the private data AnaCredit (“ analytical credit data sets ”), which contains harmonized data of the euro area in a single database for analysis, and prudential supervi soria s. In addition, the Bank for International Settlements (BIS) collects and processes information confidential bank in collaboration with central banks and national authorities for International Data Hub.

However, surprised that some of the partnerships between private companies and public institutions receive widespread criticism to social purposes. A clear example in the case of spain was the announcement by the INE purchasing data on the telephone companies geolocation from clients to explore the urban mobility. When it announced the study the media graduated very critically,: “ It is going to track the INE your phone 8 days: what operators were involved, and as to avoid it ”; ” The operators will be half a million euros to hand over to the INE data ”; “ the INE is going to spy on your mobile 8 days: what can you do. ” this reaction is surprising especially when the majority of users of a mobile phone are prepared to provide all of your personal information giving access to all the data of your phone to install a mere application of a lantern. However, if a public institution for a study with user fully anonymized telephony and aggregated by areas, set up in arms. In this regard clearly lacking in raising public awareness about the importance of the massive databases for social research.

The adoption of the rules of procedure of the RGPD (Reglamento General de protección de Datos) of the eu, adopted in 2016 and implementation since the 25 may 2018 , if interpreted narrowly sends a clear risk to the openness and cooperation of users and producers of statistics. In fact, can provide the perfect excuse those producers who want to reduce the risk to zero without taking into account the benefits of research In general the transposition of the regulation in individual countries is placing a interpretation more restrictive than the originally raised. And this despite all the safeguards that the RGPD introduced in the case of the investigation. Some examples:

( 26 ) … thus, the present regulation does not relate to the treatment of this anonymous information, even for statistical and research.

( 50 ) … subsequent treatment for file in the public interest, scientific research purposes or historically or statistical purposes should be considered processing operations compatible and lawful conditions.

( 53 ) The special categories of personal data that deserve greater protection should be treated with purposes related to health only when necessary for achieving these objectives in the interests of individuals and society as a whole … or archival purposes in the public interest, scientific research purposes or historically or statistical purposes.

The recent report Draghi, in its call to the simplification and harmonization of eu regulation in order not to damage in addition to the european economy, had many call on the impact on the investigation of the lack of harmonization of GDPR.

“ regulation of RGPD by member states and the lack of consistency in its application increase the administrative burden for eu companies. The RGPD, which entered into force 2016 and is directly applicable in all member states, aims to provide a harmonized approach of the eu for the implementation of the privacy. However, gives individual Member states the opportunity to define standards of privacy 15 Areas, resulting in fragmentation and legal uncertainty resulting from the widespread use of terms of specification, regulation and the inconsistent implementation by the data protection authorities (APD), and the fact that some member states have several APD do so (for example, 16 in Germany). This could hinder entrepreneurship and innovation, including the development and implementation of new technologies and solutions of cybersecurity. Estimates are high costs of compliance with the RGPD up 500 . 000 euros for smes and to 10 millions of euros for the large organisations. Moreover, due to costs of compliance, eu businesses reduced data storage in a 26 % and processing of data in a 15 % under comparable american companies. However, in december 2023 , member states in the formation of the council of justice and home affairs have been reluctant to greater harmonisation. ”

An important aspect on the use of data and impediments in some interpretations of RGPD is what the report Draghi called European Area of health data (EHDS). On this point the report states:

“ there is a significant potential to tap unexploited health data in the eu, as demonstrated by the considerable potential for access and link data sets in the area of health care in connection with the usa. Currently, RGPD allows the processing of health data for the provision of care or social origin, public health and scientific purposes, based in eu legislation or national origin. The data can be processed without explicit obligatory whenever appropriate and specific measures to safeguard the rights and freedoms. Some member states already benefit from these possibilities by virtue of its own national legislation. However, the adoption of these options by member states has been uneven and has led to a secondary use inefficient health data. To overcome this challenge, the commission has proposed a regulation to enable a european area of health data (EHDS) exploiting the potential offered by the RGPD for a specific law of the eu with safeguards individuals. In spring 2024 the european parliament and the council reached political agreement on the proposed regulation. The proposal aims to develop a european framework based on the measures taken by several member states that have adopted national laws similar to the secondary use of health data. ”

The report also proposes Draghi force implementation harmonized and simplified GDPR from the eu and remove regulatory overlaps in the case in the development of applications of artificial intelligence, recommending the use of a review procedure regulate and fast rules relating to the IA. The report notes that, while the ambitions of RGPD and the law of IA of the eu are commendable, “ its complexity and the risk of overlaps and inconsistencies may undermine the progress in the field of IA by industrial players of the eu. The differences between member states in the implementation of and compliance with RGPD, and the overlaps and Areas of possible inconsistency with the provisions of the law of IA create a risk that european companies are excluded from the early innovations in IA, owing to the uncertainty of the regulatory frameworks, as well as greater burdens for researchers and innovators of the eu to develop IA. ” The report Draghi pointed to the need to “ develop simplified rules and enforce the implementation of harmonized RGPD in member states, were removed with regulatory overlaps the IA. This would ensure that eu companies are not criminalized in the development and adoption of IA. ”

The use of data and algorithms of automatic learning are becoming a stronger role in economic research Collaboration between public institutions may promote the use and merger of the huge administrative databases that would increase the capacity to undertake detailed economic research and innovative. For its part the increasing participation of the private sector in the generation of useful data for economic research (assessments of public policies, monitoring of the economy to high frequency, etc.) becomes more important público-privada collaboration in the use of these databases. In this context, access to banking microdata provides one of the sources of information with the greatest potential. That is clear from many recent studies that use this type of data to analyse, with great granularity and high frequency, very important economic phenomena such as the impact of the pandemic of COVID- 19 or the impact of policies to mitigate their effects.

Until the last few years the use for the evaluation of public policies of administrative data of the spanish public institutions has been very far away from the tremendous progress that was occurring in other countries. In recent times there is a greater awareness of public institutions on the importance of administrative data for economic research, although there was still a long way to go on raising public awareness and the media. Although we have been quite so far behind other countries these initiatives offer a window of opportunity, depending on how they are established, could produce a very significant development in the quality and relevance of economic research in Spain.

AIREF ( 2020 ), Opinion for a strategy of access to administrative dataOpinion 1 / 20 , september.

Aspachs, O., Durante, R., García-Montalvo, J., Graziano, A., Mestres J., y M. Reynal (), “Real-Time Inequality and the Welfare State in Motion: Evidence from COVID-in Spain, ” Economic PolicyVolume 37 Issue 109 , 165 – 199 .

Aspachs, O., Durante, R., García-Montalvo, J., Graziano, A., Mestres J., y M. Reynal (), “ Tracking the impact of COVID-on economic inequality at high frequency, ”(),PLoS ONE, 2021

Card, D., Chetty, R., Feldstein, M., and Saez, E. ( 2010 ), Expanding Access to Administrative Data for Research in the United States. en: Schultze, C. L., and Newlon, D. (eds), Ten years and Beyond: Economists Answer NSF’s Call for Long-Term Research Agendas. National Science Foundation

Chetty, R., Friedman, J. Stepner, M. y the Opportunity Insights Team ( 2023 ), “ The economic impact of COVID- 19 : evidence from a new public dataset built using private sector data,” Quarterly Journal of Economics, 1 - 61 .

García Montalvo, J. ( 2021 a), “ economic research and data: markets, massive social purposes and collaboration público-privada, ” Economic notebooks ICE, 102 ( 2 ), 69 - 85 .

García Montalvo, J. ( 2021 (b), “ Big data and economy in real time: the usefulness of the banking microdata, ” Report of The Economic and financial EsadePol# 29 , 20 - 26 , 2021 .

García Montalvo, J. ( 2021 (c), “ Data science and its economic applications: a personal perspective ”, Chapter 1 in peña, Poncela and Ruíz (Eds.),Econometric analysis and Big Data, 2021 .

García Montalvo, J. ( 2014 ), “ the impact of the big data in the financial services, ” Papeles de economía española, 43 - 59 .

García Montalvo, J., García, A. and G. Mora ( 2007 ), “The Employment situation of Spanish university graduates some four years after graduation,” en U. Teichler (Ed.), Careers of University Graduates, 2007 , Springer Publisher, 103 - 121 .

García Montalvo, J. ( 2001 ), Education and employment of graduates in europe and in spainBancaja, Foundation, 340 pages.

OpenSAFELY Collaborative et al. ( 2020 ), “ OpenSAFELY: factos associated with COVID- 19 -related hospital death in the linked electronic health records of 17 million adult NHS patients, ” Nature, July.

Teichler, U. (Ed.) ( 2007 ), Careers of University Graduates, Springer Publisher, 103 - 121 .