Exploring the End-to-End Data Provider Journey
The Data Provider Journey focuses on ensuring that datasets are structured, findable, documented, and shared under the right conditions, to enable Data Users to create impact from reusing existing data. This journey is the second part of the End-to-End Data Journey – a framework to accelerate data access, foster collaboration, and ensure secure and responsible data sharing between Data Users and Data Providers.
In the Data User Journey, we explored how researchers, analysts, and policymakers access and analyse data for creating innovation and impact. Now, it is time to look at the data sharing obligations of data providers, also known as data holders. In this article, we explore the six steps of the Data Provider Journey from data collection to reviewing results of data reuse, portraying how data holders facilitate responsible and efficient secondary use of data.
To illustrate the journey, we combine the explanation of each step with an example of a data provider. We continue to follow Anne, the Data Provider we presented in the introduction the the End-to-End journey. Anne is responsible for managing geographic datasets within a public administration. She is the data holder, ensuring the data are properly structured, securely stored, and oversees who may have access to the data. The End-to-End Journey guides her through the essential steps of preparing and enabling access to these datasets, ensuring that data is reused in a fully compliant manner.
Step 0: Collect
The Data Provider Journey begins with data collection through various means and sources. Data providers ensure that the collected data is accurate, complete, and well-documented. It is important to describe the purpose, methodology, and limitations of the data properly in the metadata, to ensure transparency for potential future data users. In order to collect data, there needs to be consent from the data subject and/or a legal basis for the data collection. Data protection laws and regulations, as well as ethical considerations must be respected. The legal and ethical considerations are not only important from a compliance perspective, but also build trust towards the data subjects.
Example: The Data Provider Anne and her team collects geographic data from multiple sources, including satellite imagery, through public infrastructure records and surveys. This includes datasets on population distribution, urban development, and land use classifications. Anne ensures that the data collection follows necessary regulations and guidelines, requiring researchers to obtain informed consent when necessary. However, Anne needs support with the data governance, to ensure that she structures the data according to best practices in data management. Through the Data Management and Stewardship Support service, LNDS help providers like Anne in building data management and stewardship capacities, ensuring proper management of the collected data.
Step 1: Catalogue
Once data is collected, the next step is to organise it in a way that makes it discoverable and accessible. Data providers create a structured data catalogue – or add their data to an existing catalogue –, adding detailed metadata about the dataset to describe the content, source, and conditions for use. A dataset is only valuable if potential users can find it, understand it, and determine how to use it. By clearly documenting data quality, provenance, and access conditions, providers not only organise their datasets but also build trust in the data ecosystem. A well-maintained data catalogue enhances data discovery, fosters collaboration, and creates an important connection between data providers and -users. Without proper data cataloguing, valuable datasets risk being underutilised or overlooked. In other words, cataloguing is the first step to making data findable, and hence, enable the secondary use of the data.
A common misconception is to believe that a data catalogue would automatically mean that the data is accessible. It is important to note that by including data to a data catalogue only shows the metadata of the dataset, i.e. the description of what the dataset contains. The original dataset can only be accessed following strict access request procedures. To learn more on the subject, we recommend to read this article, where we answer all you need to know about data catalogues, metadata and datasets.
Example: Anne uploads her dataset into a data catalogue, ensuring that key metadata — such as study scope, data collection methods, and privacy restrictions — are documented. This allows potential data users, to find the dataset and determine whether it fits their needs. If Anne needs support with the cataloguing process, she could use the Data Cataloguing Service, which helps providers standardise metadata and improve dataset discoverability.
Step 2: Enhance
After cataloguing, the focus shifts to improving the dataset’s quality and usability. Data providers clean, standardise, and format the data to ensure consistency and compliance with established standards. This step may also involve enriching the dataset by adding annotations, linking related datasets, or integrating additional variables to make it more valuable for data users. Enhancing the data ensures that the data is well structured, consistent and well documented, and through that, easy to reuse and combine with other data. In other words, enhancing data is making it ready for analysis. By refining datasets at this stage, data providers create a smoother and more efficient process for data users while maximising the impact potential of the data.
Example: Anne’s dataset originally contained inconsistent formatting and missing values. She works with her team to standardise the structure, ensuring that her own team or other data users can easily merge it with other datasets. Anne uses support of the Data Quality and Curation service, to refine and standardise her datasets, making them more usable and reliable for current and future data users, such as the researcher Marc whom we met in our last article.
Step 3: Decide on access
At this step, the data provider has received a request for access to their data set. A data user has found the dataset and would like to use it for their (research) project. The data provider will now evaluate access request, ensuring that data user’s request and purpose of data use, aligns with the established usage policies, and comply with data protection regulations. This decision-making process is crucial for balancing accessibility with responsibility. Depending on the defined access conditions and approval processes, the data request may be evaluated by a data access evaluation committee, before providing the access to the data user.
Example: Anne receives a data access request from Marc, a researcher studying energy consumption patterns. He wants to use her dataset on land use and population density to assess energy needs in different regions. Anne ensures that Marc has submitted a clear project proposal, showcasing the exact use case and purpose of why he needs to access the dataset. Then she takes the decision forward to the data access evaluation committee. To support defining and documenting the request review process, Anne uses help of the Data Access Request and Review service.
Step 4: Provide access
Once access decisions are made, the data provider ensures that the data user can securely access the approved datasets. This involves setting up a Secure Processing Environment, configuring user permissions, and providing tools for analysis. Data providers also monitor data usage to confirm compliance with access agreements and ensure that privacy and security standards are complied with throughout the process. By facilitating secure and controlled access, data providers enable data users to focus on generating insights while maintaining full oversight of sensitive data.
Example: The data access evaluation committee has approved Marc’s request to access the dataset. Before granting access, Anne ensures that any personally identifiable or sensitive data is pseudonymised. For pseudonymisation, she could rely on the Identifier-matching and Pseudonym Management Service to ensure proper data protection, while maintaining usability for the researcher. Now, she can grant access to Marc to use the dataset within a Secure Processing Environment. This ensures that Marc can analyse land use patterns and infrastructure data while maintaining data security. While Marc is conducting his analysis of the dataset within the SPE and merging it with energy consumption data to solve his research question, Anne monitors usage logs to ensure compliance with the data access agreement. Thanks to structured access provisions, the research and analysis can proceed securely and in compliance with all regulations.
Step 5: Review output
The final step in the Data Provider Journey is reviewing the results generated by researchers. Before any findings can leave the Secure Processing Environment, data providers ensure that outputs comply with data protection standards and confirm that they align with the conditions set in the data access agreement. This involves verifying that the results do not contain any sensitive or identifiable data. By acting as gatekeepers, data providers exercise their responsibility to safeguard privacy and uphold ethical standards, fostering trust and accountability in the data-sharing process.
Example: Before Marc can publish his research findings, he must submit his results for review as we saw in Step 5 of the Data User Journey. Anne’s team checks that Marc’s final research outputs do not contain any sensitive information from the original data, such as exact locations of infrastructure or private properties. Anne ensures that all data protection regulations are met and that the published findings align with the terms of the data access agreement. Thanks to the structured data-sharing processes of the earlier steps in the Data Provider Journey, Anne is happy to see that Marc’s research aligns with the conditions of the data access agreement, and allows Marc to download his results from the Secure Processing Environment and publish his work.
Summarising the End-to-End Data Provider Journey
In conclusion, the End-to-End Data Provider Journey provides a structured framework that helps data providers like Anne navigate the complexities of sharing data responsibly and efficiently. From collecting and cataloguing datasets to reviewing research outputs, each step ensures that data is discoverable, well-documented, and shared under appropriate conditions. By following this structured process, data providers can facilitate data reuse while maintaining compliance with ethical, legal, and security requirements.
Just as the Data User Journey enables researchers to access and analyse data, the Data Provider Journey ensures that the data they rely on is accurate, well-prepared, and accessible within a controlled environment. This interplay between data users and data providers is what enables secure and impactful secondary use of data. Throughout this End-to-End Data Journey, various data services can support providers in tasks such as cataloguing, managing access requests, pseudonymisation, and enabling secure processing environments. By structuring data-sharing processes effectively, providers strengthen trust, support collaboration, and ensure that valuable datasets can contribute to research, innovation, and informed decision-making.
Discover more about how our data services can help you enable responsible data sharing and research, or contact us to start a data project.