Background

Repurposing drugs for rare diseases

There are approximately 10,000 rare diseases affecting approximately 30,000,000 individuals in the U.S.A.^{1, 2} Despite significant investments in resources over the last several decades, more than 90% of rare diseases do not have a single FDA-approved therapy.³ This fact highlights the failure of traditional new drug development to overcome the unique challenges faced by rare disease research and the need for alternative approaches.

Given that many diseases share the same underlying problems, the same treatments can often be used to treat multiple diseases. While this process is faster and less expensive than new drug development and repurposed treatments are sometimes found serendipitously, a number of barriers prevent this from happening systematically. Thus, repurposing existing FDA-approved drugs to treat rare diseases is a promising approach to help patients as quickly as possible. Similarly, drugs developed for one condition but proven ineffective or unsafe and thus never approved for the intended condition can also be used in another disease. Because the basic research and preliminary safety studies on the compound have already been completed, drug repurposing and repositioning is able to skip many steps and costs. Additionally, because the repurposed drug has already been proven safe and effective in another disease, it has a higher likelihood to succeed in clinical trials, compared to new, untested compounds.⁴

For purposes of this project, the definition of drug repurposing is: "A process of research to identify potential treatments that are already FDA-approved or in development for one disease, for use in another disease by gathering data and analyzing efficacy in order to improve treatment guidelines and access". This includes the following:

• Repurposing: an FDA-approved drug used for another disease from the one that it is approved for.
• Repositioning: a drug with some safety and/or efficacy data in one disease and modifying its structure for use in another disease.
• Reformulation: a drug with some safety and/or efficacy data in one disease and modifying its method of administration or dose for use in another disease.
• Rescue: a drug that’s not FDA-approved for any diseases due to complications in either safety and/or efficacy in the intended disease and used for another disease while maintaining the same structure and method of administration.

Due to the complexity of the steps involved for each type, the ROADMAP tool was created with a focus on repurposing drugs that are on the market, however, we refer to steps relevant to repositioning, reformulation or rescue throughout the tool.

There are a number of notable examples of drugs that have been repurposed to save rare disease patient lives, such as sildenafil for pulmonary arterial hypertension (originally for erectile dysfunction)⁵, thalidomide for multiple myeloma (originally for leprosy)⁶, and sirolimus for Castleman disease (originally for transplant rejection).⁷

Given the above, there is tremendous enthusiasm and interest in how to repurpose existing drugs for rare diseases, particularly from rare disease nonprofit organizations. However, the process and steps for how to effectively repurpose treatments is not clear. Through a partnership between the Castleman Disease Collaborative Network (CDCN), Chan Zuckerberg Initiative (CZI), and Every Cure, we launched the Repurposing Of All Drugs, Mapping All Paths (ROADMAP) project to identify the paths that can be taken to repurpose drugs, highlight the roles of various stakeholders, and centralize information on how to do this most effectively.

Acronyms that may be used throughout this page and the ROADMAP tool:

• RDNP: rare disease nonprofit organization
• CDCN: Castleman Disease Collaborative Network
• SAB: Scientific Advisory Board
• MAB: Medical Advisory Board
• RFP: request for proposals
• QOL: quality of life

For your convenience, we also are providing all the information on this page in PDF form, if you have trouble accessing the information on the interactive tool. Click the link below to download the file:

Methods

The Repurposing Of All Drugs, Mapping All Paths (ROADMAP) project aims to identify the paths that can be taken to repurpose drugs, highlight the roles of various stakeholders, and centralize information on how to do this most effectively through an interactive “ROADMAP” tool. This project is supported by a grant from the Chan Zuckerberg Initiative (CZI).

In order to understand the most effective paths for rare disease drug repurposing, we needed to obtain data on the paths taken by rare disease stakeholders and the roadblocks they are facing. We began by identifying as many rare disease nonprofit organizations and potential stakeholders as possible, distributing a survey to these stakeholders, and performing in-depth interviews with a subgroup of representative stakeholders. Then, we utilized this information along with our experiences from rare disease drug repurposing to build out the ROADMAP. We have separated out this approach into the following six steps:

1. Identifying and Characterizing stakeholders
2. Survey Execution
3. Survey Data Analysis
4. Interview Execution
5. Interview Data Analysis
6. Synthesis & Tool Development

This project was submitted for review by Advarra IRB under protocol number Pro00055201 and approved on September 28, 2021. On May 14, 2022 we submitted an update to the protocol, which included our interview protocol and updated project timeline, which was approved May 25, 2022.

This project was reviewed and approved by Advarra IRB under protocol number Pro00055201.

Identifying and characterizing stakeholders

Identifying all rare disease nonprofit organizations in the US

First, we set out to build a comprehensive list of all rare disease nonprofit organizations in the US. From here we hoped to gain an understanding of how many of them support research and drug repurposing in particular. Unfortunately, this list did not exist. We assembled a team of volunteers to combine several existing lists of organizations (i.e. NORD members, Global Genes members) and to perform additional searches to find more organizations. The final compiled list contained 982 organizations.

Characterizing rare disease nonprofit organizations

Next a larger team of 70+ volunteers spent seven months (05/18/2021-12/16/2021) extracting data from the websites of these organizations, looking for:

• Whether the organization satisfied our inclusion criteria: US-based, registered 501(c)(3) nonprofit organization, focused on one or more rare diseases (one that affects less than 200,000 people in the US (2) and having an active website
• Basic organizational info: year of founding, name & contact information of founders
• Information on the organization’s programs: conference, research agenda, biobank, registry, natural history study, etc.
• Whether the rare disease has treatment guidelines, or whether it is known that it is caused by a genetic mutation
• Any mention of drug repurposing, and if so - which drug(s) were pursued
• Any mention of partnerships or collaborations with other organizations

As a result of this exercise,711 organizations were confirmed as rare disease nonprofit organizations in the US with active websites as of 12/2021. This dataset has been made available open-source here.

Gathering data on off label use from an established medical resource

We also onboarded a team of 10 extractors who had some medical expertise or medical educational background. We took a subset of 100 rare diseases from our ROADMAP survey data and conducted an extraction process of identifying how many rare diseases had off-label drugs as a part of the official treatment guidelines on a well-known and trusted electronic clinical resource tool for physicians. From 2/17/2022 to 7/14/2022 , the team extracted every drug mentioned on the rare disease treatment page regarding a specific rare disease in the list and coded the drug entry on its “context” and “status”:

• Context refers to how the drug was mentioned on the rare disease treatment page. The options were “Context: Drug is recommended or listed as potentially helpful” and “Drug listed as does not work for this disease AND/OR no longer given AND/OR not recommended.”
• Status refers to the drugs’ approval status. The options were “FDA approved for another disease”, “FDA approved for this rare disease” and “Not FDA approved for anything.”

The goal of this data extraction was to understand the current state of rare disease treatment. A commonly used statistic is that 95% of rare diseases do not have an FDA-approved treatment. As far as we know, this statistic is never cited in any original research or dataset, so it is not clear how accurate it is. Furthermore, it does not tell us how many of these rare diseases have off-label (FDA approved for another disease, but not FDA approved for that rare disease) treatments available for them. This is crucial information for us to understand the state of rare disease treatment in the US and the need for drug repurposing.

Survey execution

Survey design

We designed a comprehensive survey using the Qualtrics platform, which included sections for several different stakeholder types: (1) rare disease nonprofit organization leaders, (2) rare disease patients, (3) rare disease patients’ loved one (parent, spouse, friend, sibling, etc. of a rare disease patient), (4) physicians who treat rare diseases, (5) rare disease researchers.

Any member of the leadership team of a US-based rare disease-focused nonprofit organization was able to participate in this research project. They were then invited to directly reach out to their US-based patient, loved one, physician, and researcher network and invite them to take the survey. Since rare diseases predominantly affect children, we allowed participants under 18 to participate in the project through an adult loved one who provided informed consent and took the survey. Additionally, since many rare diseases cause physical and cognitive disabilities, we allowed those patients to also participate in the survey through a loved one, regardless of the patient’s age. Also, we included an option for patients and loved ones to participate in Spanish, in order to be as inclusive as possible.

We focused our questions on potentially unique insights from each stakeholder:

1. Rare disease nonprofit organization leaders: organizational characteristics (age, funding, staff size, etc.), level of resources (biobank, patient registry, natural history study, scientific advisory board, etc.), the rare disease state of research (availability of treatment guidelines, diagnostic criteria, ICD code, biomarkers, etc.), information on any FDA or off-label drugs being utilized, and drug repurposing experience, including challenges they’ve encountered, the support they need, and who was in their collaboration network.
2. Rare disease patients and their loved ones (both stakeholders received the same set of questions): experience with off-label drugs for their rare disease and their general attitude towards and comfort level with drug repurposing.
3. Physicians: the frequency of and ​factors that are taken into consideration when deciding to prescribe a drug off-label, off-label drug prescription practices for the rare diseases they treat, and levels of involvement in clinical trials for rare diseases.
4. Researchers: involvement in research projects to advance drug repurposing for any rare diseases, what their motivations are, and what challenges they’ve encountered.

Preliminary Interviews

We conducted 10 preliminary semi-structured interviews with a selection of rare disease nonprofits, in order to receive feedback to refine the survey and project communication materials.

Survey Dissemination

We distributed the survey to all US-based rare disease organizations identified previously. Since this project is supported by a grant from the Chan Zuckerberg Initiative (CZI), the survey was also disseminated by the CZI leadership team to the organizations as a part of their Rare As One initiative.

The survey was officially launched on 9/29/2021 and stayed open for data collection until 1/6/2022. Because the survey included several sections and many of the respondents represented multiple stakeholders and were able to opt into multiple sections at once, we opted to include incomplete surveys at the end of the data collection period in order to gather the most data possible. In total, we received 1,923 total survey responses (completes and incompletes), which, after data cleaning and deduplication, included 723 total respondents (605 unique): 147 organization representatives, 340 patients, 170 loved ones, 23 physicians, and 43 researchers. Based on our search for rare disease nonprofit organizations, there are 711 active rare disease nonprofit organizations in the US. This means that we gathered data from approximately 20.7% of the total sample, which is notable. Though there is an obvious selection bias to organizations who are interested or pursuing drug repurposing to take our survey, review of the characteristics of these organizations suggests that we captured a variety of organizations.

Targeted Follow-up Survey

Of note, an error in the Qualtrics platform was identified in which a significant section of the patient/loved one survey was not shown to any participants. To address this, a follow up survey with these questions was sent to the 344 participants who consented to follow up. Among the 200 that responded, we were able to successfully match 191 to their original responses (See Merging Procedure below) and 156 met all inclusion criteria to be included in the full dataset for analysis.

Survey data analysis

The ROADMAP team did extensive data cleaning and analysis of the survey data, primarily using R studio⁸ packages: tidyverse,⁹ stringr,¹⁰ dplyr,¹¹ and googlesheets4.¹²

Inclusion criteria implementation & data cleaning

The data cleaning process consisted of five steps, which were executed concurrently:

1. Implementing inclusion criteria
2. Deduplication and merging
3. Name cleaning and translations
4. (Re-)classification and categorization
5. Data changes, corrections, and updates

1. Implementing inclusion criteria

Survey entries were removed from the final dataset for several reasons, including:

• Organization was not a registered 501c3 nonprofit (3)
• Organization does not support a rare disease (2)
• Organization did not meet US location criteria (155)
• Data submitted to Qualtrics after completion deadline (6)
• Progress less than 10% survey completion (399)
• Did not consent / did not answer consent question (572)
• Did not answer any questions associated with stakeholder sections (53)
• Data was a duplicate marked for removal (see “Deduplication and merging” for details) (10, not including RDNPs)

We applied each inclusion/exclusion criteria in the order presented here and updated the numbers of removals that were made after each step was applied to the dataset. This means that survey entries from later steps (e.g., “Didn’t meet US location criteria”) do not include entries from earlier steps (e.g., progress less than 10%).

In total,1,324 entries (out of an original 1,929) were removed due to not meeting inclusion criteria leaving 605 entries that qualified to remain in the final dataset.

2. Deduplication and merging

Deduplication procedure

In some cases, the same individual completed the survey twice. To identify these cases, we looked for duplicate email addresses and/or IP addresses, as there was no other identifiable data collected via the survey. Then, we identified mismatches between their survey responses, if any, and manually reconciled these differences by combining information to make entries as complete as possible. We also resolved any contradictions and inconsistencies between entries that we identified.

There were some cases where multiple rare disease nonprofit representatives completed the survey for the same organization, causing this deduplication process to be more complex. To identify these cases, we looked for rare disease nonprofit name matches among those who opted into the associated section of the survey. Then, we compared the completion rates on these entries; entries with higher progress were automatically selected over entries with lower progress. In cases where the completion rate of each entry was equal (e.g., 100%), we compared the titles of each representative. In cases where one entry’s representative had seniority (founder, CEO, executive director), we kept that entry. In a select handful of cases where both the progress and participant hierarchical standing were equal, we followed the same process described prior and manually reconciled any differences identified.

Merging procedure

The merging procedure involved combining the data from the follow up survey with the original survey data as described in a prior section. We began with 200 follow-up survey entries, and we successfully matched 191 of these entries to their corresponding ROADMAP survey entries. To do so, we compared the IP and email addresses from the original and follow-up surveys, and we merged entries where matches were found. We then isolated the data on which drugs patients from each survey reported taking. We compared groupings of drugs, and combined entries where unique matching pairs were found. Ultimately, we discarded 9 follow-up survey entries that could not be matched to the original survey data via IP/email addresses or drug names.

It should be noted that while 191 successful matches were made, only 156 entries were ultimately combined with their original ROADMAP survey data. This is because 35 of the follow-up survey entries were from respondents whose entries were removed from the dataset during the inclusion filtering process outlined in the “Filtering based on inclusion criteria” and “Deduplication and merging” sections.

3. Name cleaning and translations

Name cleaning

We reviewed and cleaned all manually-inputted job titles, specialties, organization names, disease names, and drug names. While each process varied in its scale (whereas there were only a few unique specialties to clean, there were hundreds of drug names to clean) and scope (e.g., drug/disease name cleaning efforts were extended to additional projects), the actual cleaning process was nearly identical across each cleaning activity.

Each step in the cleaning process required isolating unique names, and then developing a “key-pair” dictionary with a column for the original name, and a column for its (optional) replacement. Members of the ROADMAP team, including volunteers, would then review the original names, identify inaccuracies and inconsistencies, and generate a final list of accurate replacements. In some cases, data was marked for removal (e.g., if a disease listed was not rare, or a drug listed as FDA-approved turned out to not be FDA-approved). Any use of brand-name drugs was replaced with their generic name, and the names of pharmaceutical salts were removed as well. Acronyms were avoided whenever possible. Each dictionary was imported into R and used to clean the corresponding survey data.

Translations

Two participants who met our inclusion criteria completed the Spanish-language version of the survey. To ensure that their data would be represented in our analysis, we translated their survey responses into English. In cases where translations of participant-entered text were required, two Spanish speakers (one native speaker, and one non-native speaker) reviewed each translation for accuracy.

4. (Re-)classification and categorization

For the purposes of analysis and visualization, we created new variables that were used to classify/categorize existing data. Examples include, but are not limited to the following:

• Creating new categories for existing questions based on responses provided in the “other” sections
• Classifying disease names
• Classifying drug repurposing stages, status, and progress
• Classifying job titles
• Creating new variables (e.g., indicating whether drugs are FDA-approved based on human review; indicating whether an organization is focused on multiple diseases)

Additionally, some response categories were renamed, merged, and/or removed based on team expertise and consensus.

5. Data changes, corrections, and updates

We have made changes, corrections, and updates to our data on an ongoing, case-by-case basis. While we made minor updates to our patient, loved one, researcher, and physician data, most of our updates have been made to our rare disease nonprofit survey data. For example, in the rare case when a drug status change was identified among our list of 147 rare disease nonprofits since (e.g., a new drug being targeted for repurposing; a drug receiving FDA approval), we updated all survey question data corresponding to that drug. In a few instances where participant clarification was required to resolve inconsistencies, we reached out directly to rare disease nonprofits for their assistance using the email addresses that they provided. In some of these cases - such as when the participants directly let us know that the drug was listed in error - we would remove all data related to that drug from our dataset. Except for these major changes, we did not make updates based on any other status updates that were discovered throughout the research and interview process, and the dataset provides a snapshot of the state of drug repurposing as of survey date completion and some of the data may be or soon will be outdated.

Data analysis

The survey data analysis process mostly focused on:

1. Descriptive statistics
2. Categorization
3. Visualization

1. Descriptive statistics

We performed descriptive statistics analysis to report basic quantitative statistical data about the ROADMAP survey. Examples include frequencies, measures of central tendency, and correlations. Generally speaking, we summarized raw survey data in our descriptive analysis. In a handful of cases, however, we developed new variables and/or created aggregations of variables for the purposes of analysis. All descriptive statistics have been calculated using R and R studio(8). The overarching goal of this descriptive analysis has been to characterize the rare disease nonprofits who participated in the ROADMAP survey (their typologies; outcomes and endpoints; characteristics; resources; etc.). Below, we review the general process that we followed to visualize, report, and share our findings from the ROADMAP survey. We also explore in more detail how we used the descriptive data that we collected to answer essential questions about the drug repurposing process and the progress of RDNPs in their drug repurposing journeys.

Most of the descriptive data that we generated were presented through bar charts, pie charts, and word clouds, among other visualizations. We created rough drafts of these visualizations using R. We generally focused on showcasing aggregated frequencies of survey options. In certain cases, multiple questions were combined and aggregated; in other cases, individual questions were broken down into multiple charts. Once we narrowed our selection of visualizations and created summaries of our findings, we built out a set of interactive web pages showcasing these final visualizations and the insights gathered from them. The ROADMAP Project’s Survey Insights, available to explore here, are one of the ways that rare disease nonprofit organizations, as well as any other interested stakeholders, can explore our dataset and use it to inform further research or generate new insights.

Drug Repurposing

In order to analyze the progress of rare disease nonprofits in the drug repurposing process, we categorized the various steps involved in this process into stages. The stages that we developed were based on several questions from the survey and included:

• Early stage: Securing funding, trying to secure researcher interest to pursue, testing existing drugs in mouse or other animal models, patient data collection
• Clinical trials: Recruiting patients for clinical trials, executing clinical trials stages I, II, III
• Late stage: Analyzing clinical trial data, submitting for FDA approval, receiving FDA Breakthrough designation

Additionally, we worked to develop several success outcome categories, which would include more than just official FDA approval and provide alternative ways to conceptualize and measure drug repurposing success for rare diseases:

• FDA approval
• Off-label use, with some subjective measure of benefit (Drug to provide significant reduction in symptoms, improvement in quality of life, increase life expectancy/decrease in mortality, cure of disease, prevention of relapse)
• Off-label use, without any subjective measure of benefit
• Unsuccessful (project abandoned or FDA approval denied)

Once we had identified and classified these stages, we began to formulate specific questions about our data. While our general interest was to assess the relationship between organizational characteristics, drug identification methods, and stage(s) of progress, specific research questions included (but were not limited to):

• How many organizations had how many drugs in what stages currently?
• How each drug was identified and what stage is it in currently?
• How/Whether each organization's characteristics are related to the drug identification method used?
• How/Whether the drug identification method is related to the various types of “success” endpoints?

To answer these questions, we developed cross-tabulations of our data, which involved breaking the data into subgroups in order to look for patterns, trends, or other noteworthy observations. To be clear, this was a purely descriptive undertaking: we did not perform any kind of inferential or predictive statistics using our data. Instead, we focused on reporting raw frequencies, proportions, and percentages, and we used these numbers to help describe, characterize, and summarize our drug repurposing data, as well as answer the specific research questions outlined above.

One challenge that we had to address in developing cross-tabulations was that we were asking questions about two different units of analysis: in one case, drugs being repurposed ( 94 total); in another case, the organizations involved in drug repurposing ( 40 total). To address this, we made two separate sets of cross-tabulations, each taking a different unit of analysis. Both sets of cross-tabulations included the aforementioned list of drug repurposing stages (early, clinical trials, late, etc.). When taking repurposed drugs as our unit of analysis, we summarized this information on a per-drug basis. When taking organizations as our unit of analysis, we summarized this information on a per-organization basis. This required “flattening” the data of organizations repurposing multiple drugs into single yes/no responses. We, therefore, decided that when looking at this data on an organizational level, any “yes” responses from an organization’s survey data would overwrite any “no” responses, as our goal here was to understand whether they had reached a given stage at any point; we relied on the drug repurposing cross-tabulations for more fine-grained information about how many specific drug repurposing projects reached each stage.

When taking organizations as our unit of analysis, we compared our stage data with our data on organizational characteristics/resources. When taking repurposed drugs as our unit of analysis, we compared our data on stages with our data on drug identification methods. In the table below, drug repurposing stages are represented in the rows, while drug identification methods are represented in the columns. Each stage / drug identification method is broken into a “yes” and “no” subcategory, where “yes” indicates that a survey-taker explicitly answered “yes” to the corresponding survey question(s), and vice versa for “no” selections. We filtered out cases where the question was left completely blank (instead of inferring, for example, that this indicates a “no”), meaning that our marginal totals (the sum of all cells in a given 2x2 contingency table) do not always neatly sum to 94/40.

Each set of 4 gray highlighted squares in the snapshot below represents a single 2x2 contingency table that we could study individually and/or compare against other 2x2 contingency tables.

For a concrete example of how the above contingency table is read, we can “zoom in” on the top-left 2x2 contingency table, which displays information about two subgroups of drugs being repurposed: those in the “early” stage of drug repurposing and those that relied on “medical data” in the drug identification process. Looking at the below table tells us the following information about the relationship between this specific stage and drug identification method:

• 6 repurposed drugs are currently in the early stage of drug repurposing and were identified using medical data
• 19 repurposed drugs are currently in the early stage of repurposing but were not identified using medical data
• 14 repurposed drugs were identified using medical data, but are not in the early stage of repurposing
• 34 repurposed drugs were not identified using medical data, and are not in the early stage of repurposing

2. Categorization

One of the important outcomes of the ROADMAP project was a comprehensive understanding of all the “menu” items in the drug repurposing process, namely all the steps involved and all the options that a researcher can pursue or a representative of a rare disease nonprofit can support. Based on our experience and expertise, the ROADMAP team implemented a preliminary list of options in the survey, but via user feedback from various stakeholders, we were able to build out a more detailed breakdown of both the stages involved in the drug repurposing process as well as the options that are available for selection within each step. This categorization was done through an ongoing manual, collaborative visualization exercise using the tool MIRO , which allows for mind mapping, timeline building and other visualization techniques in an infinite, collaborative workspace. We also utilized this approach to merge the interview insights with the existing survey data, which we will describe in a later section.

3. Network Visualization

One of the sections of the rare disease nonprofit organization representative survey focused exclusively on their collaboration network. In this section, the rare disease nonprofit organization representatives were asked to list their top collaborator organizations by name, and then specify what kind of collaboration activities they were engaged in with them. From this data, we also built a network dataset of nodes (organizations) and edges (the relationships between organizations). We visualized this network using a tool called Gephi.¹³ This allowed us to gain a better understanding of the types of relationships going on between different rare disease nonprofit organizations, as well as their external links to research institutions, pharmaceutical companies and government organizations.

Interview execution

From the 147 rare disease nonprofit organizations in our survey data, we developed a typology of organizations based on various internal characteristics and drug repurposing experience.

Based on the typology, we selected 32 rare disease nonprofit organizations for participation in the interview phase of the project. We selected all 15 rare disease nonprofit organizations which selected at least one subjective success endpoint to have been met, and 17 organizations where no subjective success endpoints have been met yet. Out of these 17, we selected a sample from each stage of the repurposing process, and organizations that were comparable by other characteristics (age, funding level, staff size, etc.) We did comparisons of our selected organizations both internally (15 vs 17 organizations) as well as comparing them to the total 147, as well as those we selected for interviews vs those not selected (32 vs 115). Overall, we were satisfied with our selection, that it captured a good variability across all important characteristics.

We then invited a member of the leadership team of all 32 organizations to participate via email, and 25 agreed to participate and were interviewed. The ROADMAP project lead conducted all 25 60-minute semi-structured interviews with them over the course of two months via ZOOM. The interview protocol focused on questions regarding their experience of drug repurposing, based on a preliminary framework consisting of four stages: 1. Preparation, 2. Drug Identification, 3. Validations, 4. Clinical Use and 5. Optimal Utilization. Within each step, we generally discussed how they executed each step, how long each step took, how the decision was made as to what approach to take, what challenges they faced, what advice they have for other organizations, who they collaborated with, and what resources they utilized. In cases when organizations were in the process of repurposing multiple drugs, the interviewees were asked to discuss the ones that were either the farthest along in the process or with which they had the most varied experiences so that on aggregate, the ROADMAP would contain insights from as many varied experiences as possible. Because of this, even though 25 organizations were interviewed, we were able to capture the experiences of repurposing 75 drugs.

Interview data analysis

The interview audio was transcribed and cleaned utilizing Otter.ai, with a final manual review from research assistants and project lead to guarantee the accuracy of transcript data. Then, the project team utilized a tool called Dedoose to conduct an iterative thematic analysis of the interview transcripts.¹⁴ The project lead identified and coded statements from the interview transcript that answered a research question and generated a list of themes from these codes; this resulted in 120 themes. The project lead and research assistant then synthesized the themes into five categories of findings: 1. General Description, 2. Insights, 3. Roadblocks, 4. Support Options and 5. Resources. We then utilized a mind mapping software MIRO, as mentioned in previous sections, to identify and categorize data visually and collaboratively. This process was crucial in making sure our interview insights helped fill in any missing categories from our survey data, and provide additional advice or nuance that we were not able to capture via survey.

Synthesis and tool development

The qualitative insights from the interviews were then combined with the survey data and integrated into an interactive "ROADMAP" tool and data explorer interface. One of the most important aspects of this project is that it can help the rare disease community and that this data will be made available.

Thus, the ROADMAP project has produced several deliverables:

• The interactive data explorer
• The interactive ROADMAP tool
• Datasets
• Report
• Publication(s)

The interactive ROADMAP tool
The ROADMAP tool was developed using Jekyll, a static site generator that includes built-in hosting support for GitHub Pages. The decision to use Jekyll and GitHub pages was a tactical one: in addition to the user-friendly nature of both tools, we wanted to ensure that our tool – its code, documentation, etc. – was made available to users open-source, something that GitHub Pages readily supports. Within the Jekyll build environment, the structure of our tool was coded overwhelmingly with basic HTML/CSS/JavaScript, though we did occasionally use Liquid for the purpose of templating/automating. The content that populates our tool was written primarily using HTML, with the support of Twitter’s Bootstrap 5 framework. The ROADMAP team is also appreciative of the Codepen community, whose open-source visualizations provided fruitful starting points for the development of various “proof of concept” iterations that were designed to test the use of JavaScript to support different interactivity features (e.g., scrolling, click-throughs, etc.).

Overview of findings

The survey and our additional crowdsourced data provided a lot of unique insights from a variety of rare disease stakeholders. We are currently in the process of writing and submitting a publication with our research findings. Once this publication is available open-source, it will be available to read here. Until then, we provide preliminary, high-level findings below from each of the types of datasets we gathered as a part of this project. The data can be explored in our survey insights page here.

State of rare disease nonprofits in the US & drug repurposing

From our crowdsourcing initiative, 711 rare disease nonprofit organizations were identified in the US and basic information was extracted from organizations’ websites. Interestingly, 56% of these organizations were founded between 2011-2021. 430 (60.5%) organizations focus on a single rare disease and its subtypes, while 264 focus on multiple rare diseases (some organizations were very broad, focusing on either all rare diseases or both rare and common diseases). 416 (41.9%) of the organizations were started by loved ones of patients (parents, spouses, siblings, etc.) and 206 (20.8%) were started by the patients themselves.

Utilizing the organization’s websites, our team of extractors identified how many stated that they utilize certain resources that may be important for an organization to be able to successfully support research projects in the future, such as: 423 (59.5%) organizations had a scientific advisory board, 335 (47%) had a formalized research agenda, 291 (41%) had a natural history study or registry, 234 (33%) mentioned formalized treatment guidelines for their rare disease of focus, and 142 (20%) had a biobank.

Interestingly for this project, drug repurposing initiatives, off-label use or treatment guidelines that included repurposed drugs were found on 135 (19%) organizations’ websites.

The team of extractors extracted a total of 1,054 (542 unique) drugs for 68 rare diseases (while we started with 100 diseases, this number reduced due to combination of subtypes for the same rare disease and elimination of diseases which do not meet the official criteria for being considered “rare”). 926 drugs were listed as recommended or potentially helpful, 53 were not recommended. 702 (71.7%) were FDA approved for the rare disease in question, 203 (20.7%) were being utilized off-label, and 74 (7.6%) were not yet approved for anything in the US. Per rare disease, 304 (25.7%) unique drugs were recommended but either off label or not yet approved for anything in the US, which illustrates just how common off label drug use is and how huge the potential for drug repurposing is.

Survey insights: Rare Disease Nonprofits

The state of drug repurposing projects, led by rare disease nonprofit organizations

Among the 147 organizations surveyed, 127 support research. 58 organizations are currently pursuing drug repurposing projects and another 58 are not yet, but are interested in doing so. Out of the 58 which are pursuing drug repurposing, only 40 organizations provided a specific drug of focus. This may indicate that the remaining 18 organizations are in preliminary stages and have not yet identified a specific drug yet. Out of the 40 that already have one or more drugs for repurposing, 14 have met at least one endpoint. There is a wide range in terms of what organizations selected as their success endpoint goal: “Drug to provide significant reduction in symptoms” (48 selected this endpoint - 17 reported that is has been met), “Drug to provide significant improvement in quality of life” (48 selected - 11 met), and “Drug to be freely available to patients off-label with safety / efficacy data” (47 selected - 2 met).

While 5 drugs have made it to FDA approval, only 40 (43%) of drug repurposing projects in our data even set FDA approval as their success endpoint goal. Thus, getting an FDA approval for the drug to be used in a new disease area is often less of a goal of interest than demonstrating it is effective and helping patients, which can be accomplished without an FDA approval (as long as it is approved for another disease and widely available).

Pursuit of multiple projects

Among the 58 organizations which reported to be currently pursuing or have pursued drug repurposing projects in the past, 19 organizations have pursued multiple drugs for repurposing, with an average of 3.84 projects per organization (range of 2 - 7); most organizations only pursued two projects at a time, likely due to the financial limitations of supporting multiple complex projects.

Off-label use tracking

“Off-label drug use” refers to when a drug is prescribed for a disease that it is not specifically approved for. Out of the 58 organizations that are pursuing drug repurposing projects (or have in the past), only 17 systematically track off-label drug use in their patient population. This is important to highlight since tracking off-label drug use is an important way to assess whether a repurposed drug is effective. Data on off-label drug use can also be used as a way to identify a drug for potential further repurposing research. In fact, data analysis of off-label drug use was the third most reported method for identifying a repurposed drug, helping 14 organizations to identify 25 drugs as promising for their rare diseases. Though the majority do not track off-label drug use at this time, 66% stated that they are interested in tracking this information in the future.

Identifying a promising drug for repurposing

The most common method for identifying promising drug repurposing opportunities was preclinical/translational research (66 drugs). This category can also be described as pathogenesis targeting, whereby the researcher identifies a potential problem underlying a disease (e.g., mTOR activation is increased in Castleman disease) and then matching a drug to reverse the problem (e.g., using an mTOR inhibitor to treat Castleman disease), which is then studied further in the laboratory. The other top choices were looking at drugs used in similar diseases (29), off-label use data analysis (25), high throughput drug screening (HTDS) (21) and literature review/meta-analysis (16). The least common option was machine learning/artificial intelligence approaches (2), which may speak to the novelty of this approach and its slow integration into the existing research/repurposing processes or that it is mostly being utilized without the involvement of rare disease nonprofit organizations. It is also important to mention that more than one identification method was employed to identify 57 drugs. The most common combinations of methods were: 1) HTDS and Preclinical/Translational research (11 cases); and 2) looking at drugs used in similar diseases and Preclinical/Translational (9 cases).

It is also interesting to point out that 11 drugs were identified as promising by multiple organizations using different drug identification methods. This speaks to the value of pursuing multiple avenues at once and triangulating the findings from one method with another.

Current stage of drugs being repurposed

Among the 94 (76 unique) drugs which are in the process of being repurposed by 40 organizations, we categorized the drug repurposing project one of several stages:

• Early stage: Securing funding, trying to secure researcher interest to pursue, testing existing drugs in mouse or other animal models, patient data collection
• Clinical trials: Recruiting patients for clinical trials, executing clinical trials stages I, II, III
• Late stage: Analyzing clinical trial data, submitting for FDA approval, receiving FDA Breakthrough designation

In our dataset, 25 drugs are in early stages, 28 are in clinical trials, 14 are in late stages.

Additionally, we worked to develop several success outcome categories, which would include more than just official FDA approval and provide alternative ways to conceptualize and measure drug repurposing success for rare diseases.

Our resulting outcome categories include:

• FDA approval
• Off-label use, with some subjective measure of benefit (Drug to provide significant reduction in symptoms, improvement in quality of life, increase life expectancy/decrease in mortality, cure of disease, prevention of relapse)
• Off-label use, without any subjective measure of benefit
• Unsuccessful (project abandoned or FDA approval denied)

In our dataset, 36 drugs have reached an outcome (FDA approval / off-label use / unsuccessful) of interest. Only 5 drugs have so far made it to FDA approval for their new rare disease indication but 18 are being used off-label and having some sort of positive effect on the patients despite not having official FDA approval. 13 drugs were reported to have been abandoned. No drugs in our dataset fell into the third category, off-label use, without any subjective measure of benefit. Most drugs (58) are still in process, and have not yet achieved any of the defined endpoint categories.

*Note: due to some overlap between the stage and the outcome categories, the number of drugs in the stages does not add up exactly to the ones marked as in progress in the outcome*

High-throughput drug screening

Though many people think of high-throughput drug screening (HTDS), the process whereby lots of drugs are tested on cells or other models from a given disease to see if anything brings about a change of interest, as the primary mechanism for identifying drug repurposing opportunities, it is clear from this survey that several other methods are used to identify repurposed treatments. In our data, 21 drugs were identified via high-throughput drug screens. Out of these drugs, 6 are still in early stages, 6 are in clinical trials, 5 are in late stages, and 1 has made it to FDA approval. 6 are being prescribed off-label, and 3 have been unsuccessful.

FDA-approval

Out of our 94 (76 unique) drugs, only 5 have made it to FDA approval:

1. Dupilumab // Eosinophilic diseases
2. Selumetinib // Neurofibromatosis
3. Alpelisib // CLOVES syndrome
4. Rituximab // Pemphigus, Pemphigoid
5. Sirolimus // Lymphangioleiomyomatosis

Looking closer at the 5 organizations which have had a repurposed drug make it to FDA approval, they are on average 25.8 years old (range: 11 - 44); their annual funding ranged from $100,000 to more than $5,000,000 (the most common selection was “$1,000,000 and $2,000,000”); and they have the following characteristics: all 5 have an SAB/MAB, 3 have a natural history study, 3 have a formal research agenda, 3 already have an FDA approved drug prior to pursuing drug repurposing, 2 have a patient registry, and 1 has a biobank. One of these organizations has no full-time staff, relying entirely on volunteer or part-time staff to achieve their success, while the other four have anywhere from 1 to 40 full-time staff.

The amount of time that passed between initial FDA approval and repurposing approval ranged greatly, from 3 up to 21 years. Also worth mentioning that Selumetinib was a case of drug repositioning, as it was being explored as a potential drug for several indications, but was not pursued all the way to FDA approval; instead, it was identified as promising for an alternative use and received its first FDA approval for Neurofibromatosis.

It is important to note that there are very many factors that affect whether a drug can ever get FDA approval that do not depend on its safety, efficacy and are beyond the control of a researcher or a rare disease nonprofit organization supporting a drug repurposing project. Notably the most important of these is whether the pharmaceutical company who developed the drug is interested in investing the time and money into supporting the FDA approval application for a new disease when it is already approved for another disease. Furthermore, FDA approval for the new disease is not always necessary in order to have a drug be able to reach patients in need (the primary purpose). Thus, we consider FDA approval to be one of many metrics of success.

Off-label use with some subjective measure of benefit

As an alternative success outcome to FDA approval, we can consider off-label, with some subjective measure of benefit, such as being freely available to patients off label with safety / efficacy data, providing significant reduction in symptoms, improvement in quality of life, increase life expectancy / decrease in mortality, provide cure of disease, provide prevention of relapse. If we look at organizations that fit this criteria for at least one drug, we end up with 12 organizations. They are on average 17.6 years old (range: 2 - 44); the majority (4 organizations, 33.33%) reported annual funding between $100,000 and $500,000; and they have the following characteristics: 11 (91.67%) have an SAB, 6 (50%) have a natural history study, 7 (58.33%) have a formal research agenda, 6 (50%) have a patient registry, 5 (41.67%) have a biobank, and 3 (25%) already have an FDA-approved drug prior to pursuing drug repurposing (one organization has two FDA-approved drugs). Interestingly, 6 (50%) have no full-time staff, relying entirely on volunteer and/or part-time labor to achieve their success.

Among these 12 organizations, there are 44 unique drugs being repurposed. The most common drugs among these organizations were Sirolimus (3 organizations), Trametinib, Everolimus, and Bevacizumab (2 organizations each). The most common identification method for these drugs was Preclinical/Translational research (30), closely followed by data from similar diseases (17) and off-label use (17). Most of these drugs are currently in early stages or clinical trials, specifically in recruiting patients for clinical trials (18). Their respective rare diseases have the following characteristics: 11 (91.67%) have animal models, 8 (66.67%) have cell lines developed, 9 (75%) have an identified genetic mutation, 8 (66.67%) have an ICD code, 7 (58.33%) have treatment guidelines, 5 (41.67%) have a clear understanding of etiology or disease pathogenesis, and 3 (25%) have predictive biomarkers.

Correlational insights

We compared data on drug identification methods with the stages of the drug repurposing process and success outcomes. In the table below, drug repurposing stages are represented in the rows, while drug identification methods are represented in the columns.

From this table, we can see that preclinical/translational research seems to be the approach most often associated with clinical or late stages of the repurposing process, and in the outcomes, it is most associated with FDA approval or the alternative success outcome, off label use with measures of success.

We also correlated the organizational characteristics with the drug repurposing stages. In the table below, drug repurposing stages are represented in the rows, while organizational resources are represented in the columns.

From this table, we can see that having an SAB and community discussion platform appear to be the factors most associated with the clinical phase of repurposing and not being unsuccessful.

Collaboration across rare diseases

It is very interesting that the second most-selected drug identification method was looking at drugs used in similar diseases , as one of our research focuses was the levels of collaboration between different organizations, and we were specifically interested to see how much collaboration we would see among different rare disease nonprofits. With the data from the collaboration section of the survey (120 out of the total 147 of nonprofits provided this info), we were able to build networks. Each node in a network represents an organization, and each edge signifying a connection between them, either an explicitly mentioned connection via the survey data or an affiliation edge which was later added by the research team. The nodes are colored by type of organization, with a distinction made between our participating rare disease nonprofit organizations (Rare Disease Non Profit (RDNP), and other nonprofit organizations mentioned by the respondents (Non profit):

From looking at the network visualization, we can see that most of the rare disease nonprofit organizations in our data are actually part of one large “connected component”. A connected component means that there is a path between every pair of nodes. In other words, it is a group of nodes that are all reachable from one another by following the edges of the graph. The nodes here are sized by degree, with the largest two nodes being NORD and Global Genes, which is not surprising since these are large, umbrella organizations that each have hundreds of member organizations.

Additionally, since we asked organizations about the types of collaborations they are engaged in with their top 5 closest collaborators, we are able to view subsets of the networks based on the type of activity. For example, we can see organizations who share data, share resources and share knowledge about their experiences (See Figure 12, in which green edges = yes, red edges = no). There is some overlap within those who come up as doing all three (consistently green edges across all three networks below), but there are some significant differences. For one, knowledge sharing seems to be a lot more common, as compared to data or resource sharing (evident by the fact that there are a lot more green edges, compared to the other two networks). Also, even though NORD and Global Genes are the highest nodes by their number of connections to other organizations overall, they were not reported to be in the top 5 organizations with which our participating nonprofits engage with data sharing or resource sharing with (evident by the fact that most of the edges in these two networks are red). This may indicate that even though these types of data sharing might still be happening, rare disease nonprofits work more closely with other nonprofit or academic institutions for these types of collaborations.

You can explore more of the rare disease nonprofit data on the ROADMAP survey data explorer .

Acknowledgements

We wish to thank the following individuals and organizations for their contributions to this important work:

CDCN Team
Dr. David Fajgenbaum, Ania Korsunska, Mileva Repasky, Mary Zuccato, Sarah E. Bolden,
and prior CDCN/CSTL team members Johnson Khor, Dr. Ruth-Anne Langan Pai, Hemi Park, and Trae Boyd

Participating rare disease nonprofit organizations
Acromegaly Community Inc, Adult Polyglucosan Body Disease Research Foundation, Aicardi-Goutieres Syndrome Americas Association, AKU Society of North America, Alport Syndrome Foundation, Amyloidosis Support Groups Inc., Angelman Syndrome Foundation, ASXL Rare Research Endowment Foundation, Autoinflammatory Alliance, Barth Syndrome Foundation, Ben's Friends Patient Communities, Beyond Batten Disease Foundation, Bow Foundation, BPAN Warriors, CACNA1A Foundation, Campaign Urging Research for Eosinophilic Disease, Cardio Facio Cutaneous International, Castleman Disease Collaborative Network, CDG CARE, Champ Foundation, CHAMP1 Research Foundation, Chelsea's Hope Lafora Children's Research Fund, Child Neurology Foundation, Children's Tumor Foundation, Cholangiocarcinoma Foundation, Chordoma Foundation, Choroideremia Research Foundation, Chronic Recurrent Multifocal Osteomyelitis Foundation, Circadian Sleep Disorders Network, CLOVES Syndrome Community, Coalition to Cure Calpain 3, Congenital Hyperinsulinism International, Costello Syndrome Family Network (CSFN), CSNK2A1 Foundation, Cure CMD, Cure GM1 Foundation, Cure HHT, Cure JM Foundation, Cure VCP Disease, Inc., cureCADASIL Association, CureSearch for Children's Cancer, Cyclic Vomiting Syndrome Association, Daphne's Lamp, DDX3X Foundation, DeSanto-Shinawi Syndrome, Dreamsickle Kids Foundation, DRESS Syndrome Foundation, Dup15q Alliance, E.WE Foundation, Emily's Entourage, Epilepsy Alliance America, Facial Pain Association, Familial Chylomicronemia Syndrome (FCS) Foundation, Fanconi Anemia Research Fund, Foundation for CAMK2 Therapeutics, Foundation for Ichthyosis and Related Skin Types, Foundation for Sarcoidosis Research, FOXG1 Research Foundation, FPIES Foundation, GACI Global, Glut1 Deficiency Foundation, Hannah's Hope Fund, Hermansky-Pudlak Syndrome Network, Hugs For Mito, Inc., Huntington's Disease Society of America, Hypersomnia Foundation, Hypoparathyroidism Association, IDefine, Indian Organization for Rare Diseases, Infantile Neuroaxonal Dystrophy Cure Foundation, International Autoimmune Encephalitis Society, International Fibrodysplasia Ossificans Progressiva Association, International Pemphigus Pemphigoid Foundation, Jamal's Helping Hands Inc., Jansen de Vries Syndrome Foundation, Jansen's Foundation, Jordan's Guardian Angels, Kabuki Syndrome Foundation, KIF1A.ORG, Koolen-de Vries Syndrome Foundation, LAM Foundation, Life Raft Group, Liv4TheCure, Lymphangiomatosis & Gorham's Disease Alliance, Malan Syndrome Foundation, Mast Cell Hope, MEPAN Foundation, Mission: Cure, Mississippi Metabolics Foundation, Muscular Dystrophy Association, Myasthenia Gravis Foundation of America, Myositis Support and Understanding Association, National Foundation for Ectodermal Dysplasias, National Neutropenia Network, National Organization for Disorders of the Corpus Callosum, National PKU News, NEC Society, NephCure Kidney International, Neurodegeneration with Brain Iron Accumulation Disorders Association, Neurofibromatosis Northeast, Ocular Melanoma Foundation, Oligo Nation, Organic Acidemia Association, Pachyonychia Congenita Project, PCDH19 Alliance, Pityriasis Rubra Pilaris (PRP) Alliance, Platelet Disorder Support Association, Primary Ciliary Dyskinesia (PCD) Foundation, Project ALS, Progressive Familial Intrahepatic Cholestasis (PFIC) Network, PSC Partners Seeking a Cure, PTEN Hamartoma Tumor Syndrome Foundation, RASopathies Network, Raymond A. Wood Foundation, Recurrent Respiratory Papillomatosis Foundation, Remember The Girls, RUNX1 Research Program, RYR-1 Foundation, Sara's Cure AKA Clear Cell Sarcoma Foundation, SATB2 Gene Foundation, Shwachman-Diamond Syndrome Foundation, Siegel Rare Neuroimmune Association, SLC6A1 Connect, Smith-Kingsmore Syndrome Foundation, Smith-Lemli-Opitz /RSH Foundation, Snyder-Robinson Foundation, Sophie's Hope Foundation, Spastic Paraplegia Foundation, Spinal CSF Leak Foundation, Stiff Person Syndrome Research Foundation, STXBP1 Foundation, Sumaira Foundation, Superficial Siderosis Research Alliance, Syngap1 Foundation, T.E.A.M. 4 Travis (Together Ending Asplenia Mortality), Tatton Brown Rahman Syndrome Community, TBCK Foundation, Team Telomere, United States Thrombotic Microangiopathy Alliance, Usher 1F Collaborative, Usher Syndrome Coalition, Vici Syndrome Foundation, Inc, Pulmonary Alveolar Proteinosis (PAP) Foundation, TESS Research Foundation, and all their participating patients, loved ones, researchers and physicians.

Volunteers
Meg Zuccato, Anna Nguyen, Derek Ansel, Martin Lukac, Annalise Jear, Mitav Nayak, Panchatapa Baul, Samatha Hood, Sabina Grigorian, Jacob Lowy, Justin Wong, Sydney Grisham, Yuan (Abby) Feng, Leanna Chen, Megan Shieh, Bryan Aguilar, Marcy Spiker, Rose Weathers, Katherine Fang, Veikko Toikka, Robert Parillo, Lindsay McBride, Sara Barrett, Robert Parillo, Dallas Ryan, Veikko Toikka, Benita Balogun, Penny Deremer, Miti Patel, Emma Roemer, Neda Pazuki, Erikka Chowdhury, Stephanie Hage, Matt Scott, Geetha Turlapati, Jessica Xiang, Jada Watkins, Jazmin Loughlin, Justin Crawmer, Ferzana Niazi, Anaheit Arathoon, Michael Zhang, Vee Suresh, Mahima Sangtani, Sara Cronin, Rita Aberbach, Kayleigh Nicole Murray, Diane Baynes, Andrew Zhu, Anastasia Kakurina, Susanna Hunanyan, Jade Bondy, Simarsukh Dhillon, Angela Perry, Carolyn Canterbury, Owen Yu and Kanan Lathia.

Partners & Collaborators
Chan Zuckerberg Initiative, Medidata Solutions

Other
Beacon, Rare Revolution Magazine, CDRC, PTEN Research

Online resources
Codepen, Icons8, Bootstrap 5, MDB 5

References

1. Haendel M, Vasilevsky N, Unni D, Bologa C, Harris N, Rehm H, et al. How many rare diseases are there? Nat Rev Drug Discov. 2020 Feb;19(2):77–8. ↩

2. Orphan Drug Act [Internet]. 96 STAT. 2049, 97–414 Jan 4, 1983. Available from: https://www.fda.gov/media/99546/download. ↩

3. Kaufmann P, Pariser AR, Austin C. From scientific discovery to treatments for rare diseases – the view from the National Center for Advancing Translational Sciences – Office of Rare Diseases Research. Orphanet Journal of Rare Diseases. 2018 Nov 6;13(1):196. ↩

4. Stone H. CURE ID: Capturing Clinician’s Experiences Repurposing Drugs to Inform Future Studies in the Eraof COVID-19 [Internet]. 2020 Jun 9 [cited 2021 Apr 5]; FDA. Available from: https://www.fda.gov/about-fda/fda-pharmacy-student-experiential-program/fda-drug-topics-cure-id-capturing-clinicians-experiences-repurposing-drugs-inform-future-studies-era. ↩

5. Ghofrani HA, Osterloh IH, Grimminger F. Sildenafil: from angina to erectile dysfunction to pulmonary hypertension and beyond. Nature Reviews Drug Discovery. 2006 Aug;5(8):689–702. ↩

6. Latif T, Chauhan N, Khan R, Moran A, Usmani SZ. Thalidomide and its analogues in the treatment of Multiple Myeloma. Experimental Hematology & Oncology. 2012 Sep 11;1(1):27. ↩

7. Fajgenbaum D, Langan RA, Japp AS, Partridge HL, Pierson SK, Singh A, et al. Identifying and targeting pathogenic PI3K/AKT/mTOR signaling in IL-6 blockade–refractory idiopathic multicentric Castleman disease. J Clin Invest. 2019;129(10):4451–63. ↩

8. RStudio Team. RStudio: Integrated Development Environment for R [Internet]. Boston, MA: RStudio, PBC.; 2020. Available from: http://www.rstudio.com/. ↩

9. Wickham H, Averick M, Bryan J, Chang W, McGowan LD, François R, et al. Welcome to the tidyverse. Journal of Open Source Software. 2019;4(43):1686. ↩

10. Wickham H. stringr: Simple, Consistent Wrappers for Common String Operations [Internet]. 2022. Available from: https://CRAN.R-project.org/package=stringr. ↩

11. Wickham H, François R, Henry L, Müller K. dplyr: A Grammar of Data Manipulation [Internet]. 2022. Available from: https://CRAN.R-project.org/package=dplyr. ↩

12. Bryan J. googlesheets4: Access Google Sheets using the Sheets API V4 [Internet]. 2022. Available from: https://CRAN.R-project.org/package=googlesheets4. ↩

13. Bastian M, Heymann S, Jacomy M. Gephi: an open source software for exploring and manipulating networks. In: Third international AAAI conference on weblogs and social media. 2009. ↩

14. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. 2006 Jan;3(2):77–101. ↩

15. Korsunska A. Research Agenda-Setting in Medicine: Shifting from a Research-centric to a Patient-centric Approach. In: Information in Contemporary Society. Springer International Publishing; 2020. ↩

16. Zuccato M, Shilling D, Fajgenbaum DC. The Collaborative Network Approach: a model for advancing patient-centric research for Castleman disease and other rare diseases. MacKenzie A, Groft S, Justice M, van Karnebeek C, editors. Emerging Topics in Life Sciences. 2019 Mar 29;3(1):97–105. ↩

17. Rao A, Vaught J, Tulskie B, Olson D, Odeh H, McLean J, et al. Critical financial challenges for biobanking: Report of a National Cancer Institute Study. Biopreservation and Biobanking. 2019;17(2):129–38. ↩

18. Acar OA, Tarakci M, van Knippenberg D. Why Constraints Are Good for Innovation. Harvard Business Review [Internet]. 2019 Nov 22 [cited 2021 Apr 30]; Available from: https://hbr.org/2019/11/why-constraints-are-good-for-innovation. ↩

19. Acar OA, Tarakci M, van Knippenberg D. Creativity and Innovation Under Constraints: A Cross-Disciplinary Integrative Review. Journal of Management. 2019 Jan 1;45(1):96–121. ↩

20. Pushpakom S, Iorio F, Eyers PA, Escott KJ, Hopper S, Wells A, et al. Drug repurposing: progress, challenges and recommendations. Nature Reviews Drug Discovery. 2019 Jan;18(1):41–58. ↩