General
Legacy ID
1
Changes to the Gross Receipts Calculation in the SBA Program Regulation
Secure Data Commons - What You Need to Know
What You Need to Know
Secure Data Commons (SDC) enables collaborative and controlled integration and analysis of research data at the moderate baseline security level. FedRAMP security impact level, modern, includes upholding security controls for personally identifiable information (PII) and confidential business information (CBI) data. SDC offers authorized and controlled access to individual datasets and its associated metadata. Additionally, SDC supports the following data transfers: real-time (streaming), batch (daily, weekly), and ad-hoc (occasional).
Considering the SDC for your project, data, or analysis needs?
Here's what you need to know.
Project Sponsor
- Conceptualizes how the SDC can support the research program's mission needs and obtains a cost estimate
- Understands the overall process and project lifecycle for using the SDC
Data Provider
- Submits research datasets hosted on the SDC
- Establishes the data protection needs and acceptable use terms for Research Analysts
Data Steward
- Manages, controls, and maintains the quality of data assets
- Appropriates protections, restrictions, and other safeguards depending on the nature of the data
- Authorizes access to data, data exports, and establishes data retention policies
Research Analyst
- Conducts complex analysis using datasets hosted in the SDC
- Brings own data and tools into the SDC
- Uses data and tools available to create meaningful insights that can be used to inform data-driven research and/or policy
Technical Requirements
SDC researchers are expected to have foundational analytical programming experience in R, Python, SQL, and GitLab, and familiarity with cloud services to access datasets within the SDC platform.
Security
Each participating data provider in the SDC defines the levels of access to SDC users. Based on the agreement with the data provider, the SDC team granularly grants access to approved users of the data.
Access to SDC Web Portal
Currently the SDC is available only to USDOT-approved projects. If you would like access, or would like to propose your research to be a part of the SDC community, please email sdc-support@dot.gov.
For more information on Secure Data Commons, please see SDC Capabilities (pdf).
Last updated November 2022
Secure Data Commons - Support
Training Materials
Series of training videos to get to know the SDC services and capabilities.
SDC 101 Overview
SDC Personas Guide
Overview of the different personas involved in the SDC including the Project Owners, Data Stewards, Data Providers, and Data Analysts.
How to Request Access to Datasets in the SDC
Inform Secure Data Commons (SDC) users how to request access to specific datasets through the SDC Web Portal.
Getting Started with DBeaver
Overview of how SDC Researchers can get started with using DBeaver.
How to Approve Data Exports out of the SDC
Overview of how SDC Data Stewards can approve data exports out of the SDC.
How to Access and Launch Your Workstations
Overview of how to access your workstations within the SDC.
How to Approve Trusted User Requests
Overview of how to approve trusted user export requests.
How to Request Trusted User Status in the SDC
Overview of submit approval to be a Trusted Researcher within the SDC to support automated exports.
SDC Vocabulary Video Part 1
SDC Vocabulary Video to inform researchers on what the SDC is and what workstations are.
SDC Vocabulary Video Part 2
SDC Vocabulary Video to inform researchers on what types of datasets that exist on the SDC.
Demo Videos
SDC Researcher Demos
Recorded video of an interactive demo of research tools and Notification Service
SDC User Guides
Research Team User Guide
User Guide for Researchers including materials on open source collaboration using GitLab.
Data Provider User Guide
For more videos and more information about the SDC please reach out to the SDC Support Desk.
Last updated January 2025
Secure Data Commons - Starting A Project
Starting a Project in the SDC for Project Owners
Managing and sharing transportation research data, securely, is now easier than ever. The USDOT Secure Data Commons (SDC) is an access-controlled, cloud-based data environment that enables users to conduct analyses and develop new tools around emerging sources of transportation data. The SDC stores transportation data made available to researchers to work with these datasets.
Cost of using the SDC
The SDC tracks project-related usage of the system and has a model to help estimate labor and cloud resource costs to support your research needs. SDC platform features that are broadly available to all participating projects, are also shared costs by all such projects.
Estimate the Cost for Your Project
Contact SDC Support to see how much your project might cost based on your need. Please email sdc-support@dot.gov.The SDC is right for you if...
Depending on the requirements of the project, the table below can help you determine if SDC is suitable for your needs.
| Questions | Yes | No |
|---|---|---|
| Does your project generate data that needs to be access-controlled ? | ✅ SDC will store and analyze sensitive data with fine-grained access control. Access is authorized to users through a data use agreement with revocable access terms to protect the sensitivity of data. | Project-level data management may be secure enough. |
| Does your project require collaboration between DOT Analysts and Researchers in the transportation community using the same data and tools? | ✅ SDC promotes team collaboration by enabling data analysis tools, custom toolsets, open code sharing, and dataset curation to data analysts and external data providers. | Data sharing and analysis for the project team, using the same tools may be sufficient. |
| Does the project generate or need to manage sensitive data? | ✅ If the project manages only moderately sensitive data, then SDC is a good option. If data is not sensitive, the SDC is not required. Public data hubs may be a better option (like the ITS Data Hub) | The SDC may not a good option for managing extremely sensitive data |
| Does your project need to combine data from a diverse set of transportation data sources? | ✅ SDC leverages cloud capabilities to share complex transportation datasets with the transportation research community to analyze problems or gain insights. | SDC will enables scalable data storage, data analysis and user access via cloud-based platforms |
Starting a Project
The SDC admin team works with the project owner to introduce the data providers and analysts to the SDC and get your project up and running.
Each SDC project goes through a standard lifecycle that prepares them for success within the SDC.
See what this process may look like for your project.
What's Next?
As a project owner interested in joining the SDC, use the steps below to get started:
Reach Out to SDC Team. Contact SDC Support to get an introductory overview of the services and platform features that SDC provides. Email sdc-support@dot.gov .
Let's Meet! During the SDC 101 Overview meeting, come prepared to provide an overview of your research program objectives, goals, and any security concerns. You will gain an understanding of SDC's capabilities, dataset availability, and customer support services.
Way Forward. SDC team will provide a Cost Estimation to support your program needs and notional timelines. Steps to fund SDC supporting services will follow the Working Capital Fund process. SDC services are flexible to exercise options to provide dedicated capacity.
Last updated May 2024
Secure Data Commons - SDC Lifecycle
SDC Lifecycle
Each project in the USDOT Secure Data Commons (SDC) goes through a standard lifecycle of activities.
Prospective
SDC team works with Project Owners who have a clear objective and a set of data providers and/or data analysts to support your research project needs.
Planned
We work with you to understand the objective, timelines, data provider and data analyst needs.The SDC and project teams collaborate to define the high-level requirements for the project and assess the suitability of the platform.
Discovery
We collaborate with you to define the detailed requirements for your project including gathering dataset documentation to demoing data pipeline architecture.
Onboarding
SDC Team implements data ingestion and curation requirements. We provide access to workstations, training materials, and assistance for queries and data uploads to support cross-project collaboration.
Active
Project team utilizes the SDC platform to achieve research objectives, while we actively provide Support Desk service.
Export
We work with you to conclude active research work in the SDC platform and begin planning retirement. (No new data coming from data providers.)
Retire
SDC Team archives project data as per records retention schedule, removes team members, and works with Project Owner to conduct final closeout.
Last updated August 2021
Secure Data Commons - Getting Data to SDC
Getting Data to SDC
Upload, store, and access your datasets quickly, easily, and securely.
The Secure Data Commons (SDC) enables collaborative but controlled integration and analysis of research data at the moderate sensitivity level, including personally identifiable information (PII) and confidential business information (CBI).
Within the SDC, you can upload data in near real time throughout your project. You can develop common data formats and fix issues as data starts to be generated.
As a data provider, you define the terms of access and grant/deny access to specific SDC users or groups. You can also control the type of derived data users can export or copy from the system.
Features That Matter
Frequent data transfers
- Real-time (e.g., streaming) data ingestion
- Batch (e.g., daily or weekly) data ingests
- Ad hoc (occasional) uploads
Cloud-based data management
- Strong security controls to export derived data out
- Set levels of data curation and warehousing to support analysis
- Validation of incoming datasets in near real time
- Real-time data ingestion streams and batch uploads
Strong access controls
- Multifactor authentication and personal identity verification (PIV) card integration
- Secure workflow for data import and export
- Controlled access to specific datasets and metadata by individual users and teams
Built-in data analysis tools
- Preestablished workstations with open source tools
- Ability to import and share code between researchers
Effective team collaboration
- Project-level controls and teams
- Shared team internal code repositories
Multiple data formats
- Raw
- Curated
- Published
What's Next?
User the steps below to bring your data into the SDC:
Contact the SDC team for a discovery meeting to discuss data restrictions, Data Provider agreement, and user access level to your data within the SDC. Review the Data Provider User Guide.
Once the data is in the SDC, work with the SDC team to monitor the quality and quantity of your data.
Our Enablement Services team offers custom upgrades to help to support your research mission needs along the way.
Last updated October 2023
Secure Data Commons - Enablement Services Program
Enablement Services Program
The Enablement Services Program provides options for you (project owners, data providers, data analysts) and your project team to optimize use of the Secure Data Commons (SDC). Project teams start with five offerings at the baseline category. You may select upgrades (silver or gold) for each offering to meet your project needs.
Still new to the SDC? You can learn more about what the SDC is.
Program Offerings
Project Onboarding and Training
Uploading Data
Data Cleanup
Analyst Setup and Collaboration
Data Analysis Consulting
Service Category Overview
Every project starts with baseline services for each offering. You may wish to upgrade to silver or gold services for additional assistance.
Baseline
Services
($)
Every project begins here, which includes:
- Overview web session
- Access to training materials
- Help for uploading data
- Analyst workstation setup
- Assistance with basic queries
- Cross-project collaboration
Silver
Services
($$)
Silver services include all baseline services, plus:
- Consultations with data analysts
- Pre-planning report
- Sessions about project costs
- Data preparation for queries
- Analytical tool installation and support
- Database optimization
Gold
Services
($$$)
With gold services, the premier option, you receive all silver services, plus:
- On-site consultations
- Building analytical tools
- In-person onboarding
- Specialized training
- Data documentation
- Automating data upload scripts
- Performance boost of analytical models
Available Program Offerings
See more information about each program offering below.
Contact the SDC team to get a quote, upgrade or change your services. If you are an independent analyst not on a project team, email us to find out how the offerings can work for you.
Email SDC TeamProject Onboarding and Training
Personal guidance and coaching about what you need to know to use the SDC
Offering Details
Baseline ($)
Silver ($$)
Gold ($$$)
Offering Details Overview web session with your team
Baseline ($)
✔Checkmark
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering DetailsDocumentation and training materials
Baseline ($)
✔Checkmark
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering DetailsDiscovery sessions about business needs and costs
Baseline ($)
✔Checkmark
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Project pre-planning assessment report
Baseline ($)
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Multiple sessions to estimate and refine project costs
Baseline ($)
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Dedicated onboarding support
Baseline ($)
Silver ($$)
Gold ($$$)
✔Checkmark
Offering Details 2-day training with your team
Baseline ($)
Silver ($$)
Gold ($$$)
✔Checkmark
Uploading Data
Working with data providers to safely upload data into the SDC, regardless of the size
Offering Details
Baseline ($)
Silver ($$)
Gold ($$$)
Offering Details Process setup, provide user guide for secure upload
Baseline ($)
✔Checkmark
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Refining or development of your external scripts
Baseline ($)
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Build metadata, dictionaries for your project
Baseline ($)
Silver ($$)
Gold ($$$)
✔Checkmark
Offering Details Software development for automating data upload
Baseline ($)
Silver ($$)
Gold ($$$)
✔Checkmark
Data Cleanup
Support to ensure quality, performance, and reliability of your data entering the SDC
Offering Details
Baseline ($)
Silver ($$)
Gold ($$$)
Offering Details Setting up data for easy queries
Baseline ($)
✔Checkmark
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Making complex datasets available to query
Baseline ($)
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Error detection and reporting during data upload
Baseline ($)
Silver ($$)
Gold ($$$)
✔Checkmark
Offering Details Building automated tools to fix errors for the upload process
Baseline ($)
Silver ($$)
Gold ($$$)
✔Checkmark
Analyst Setup and Collaboration
Help to organize and improve your analyst's research capabilities while using the SDC
Offering Details
Baseline ($)
Silver ($$)
Gold ($$$)
Offering Details Analyst workstations that come with standard tools
Baseline ($)
✔Checkmark
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Help with installation of other approved open-source software
Baseline ($)
✔Checkmark
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Setting up collaboration tools based on individual needs
Baseline ($)
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Assist install of new analytical tools on the SDC
Baseline ($)
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Create custom analyst software, up to two per year
Baseline ($)
Silver ($$)
Gold ($$$)
✔Checkmark
Data Analysis Consulting
Tailored advice, including technical support and resources, to advance the performance of your models and analytical outputs
Offering Details
Baseline ($)
Silver ($$)
Gold ($$$)
Offering Details Specific guidance on using SDC infrastructure
Baseline ($)
✔Checkmark
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Regular consultations to review challenges and SDC updates
Baseline ($)
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Database optimization for your project needs
Baseline ($)
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Reviewing and improving data scripts and queries
Baseline ($)
Silver ($$)
✔Checkmark
Gold ($$$)
✔Checkmark
Offering Details Support to boost the performance of data analyst models
Baseline ($)
Silver ($$)
Gold ($$$)
✔Checkmark
Last updated November 2021
Secure Data Commons - Current Data
Featured Data
The USDOT Secure Data Commons (SDC) platform features datasets for the following projects - check back soon as new projects (datasets) are added:
Waze for Cities Program
Federal Highway Administration (FHWA) Highway Safety Information System (HSIS)
Federal Aviation Administration Functional Genomics Research
Waze for Cities Program
The U.S. Department of Transportation's (USDOT) Safety Data Initiative (SDI) is a cross-cutting, collaborative effort within DOT, led by the Office of the Secretary of Transportation. The intent of SDI is to build on and enhance safety efforts related to data, analysis, and policy making. SDI is integrating Waze data with transportation data to develop rapid crash indicators.
Federal Aviation Administration (FAA) Functional Genomics Research
FAA Functional Genomics team uses the SDC to perform large-scale analyses of data generated from research on biological specimens, primarily derived from human subjects research. The data generated may include genetic sequence and other molecular data, physiology, demographics, study conditions, and performance metrics. This is not available to the public through the SDC at this time. FAA Genomics data owners will make subsets of data accessible in alternative locations, to the extent compatible with subject consent and Institutional Review Board concurrence as appropriate, after review and consideration of proper privacy and data quality elements. For instance, sequence data when ready may be hosted in secure-access repositories appropriate to those types of data.
FHWA Highway Safety Information System (HSIS)
The Federal Highway Administration (FHWA) developed the Highway Safety Information System (HSIS) to support safety research programs and provides input for program policy decisions. HSIS is a roadway-based system that provides quality data on a large number of crash, roadway, and traffic variables. The crash, roadway inventory, and traffic volume data are acquired annually from a select group of States.
FHWA provides this data to researchers upon request through the HSIS webpage. Educators who wish to use HSIS data for instructional purposes in a road safety course should contact HSIS staff directly at Ana.Eigen@dot.gov. For more information https://highways.dot.gov/research/safety/hsis
December 2024
Secure Data Commons - Conducting Analysis
Conducting Analysis in SDC
As a data analyst, work within the USDOT Secure Data Commons (SDC) to share code and data, upload datasets, and export approved derived analyses. Through the SDC, you can:
- Share code and data with other analysts
- Upload your own datasets
- Export approved derived analysis
We'll provide you with a cloud-based workstation with preloaded programming environments and software that grants you access to the data lake and data warehouse. The workstation also includes commercially available tools - no local software or tool installation needed!
Analytical Tools and Query Languages Supported
The SDC platform provides on-demand access to popular programming and statistical tool packages for cloud-based processing (for experienced analysts). Other, nonstandard software can be installed upon request, both individually and across user groups. For software requiring special licenses, analysts may provide their own existing licenses.
Analytical Tools
Custom options available upon request
Types of Datasets
The SDC platform provides a data lake of transportation-related structured, semi-structured, and unstructured datasets that are stored in raw, curated, and published formats. Each dataset has different data agreements based on the complexity and sensitivity of the data. Access to specific data is approved by data providers - learn more about specific dataset formats below:
Raw Datasets
Raw datasets are unaltered data are stored in their native/original "as-is" format. Uploads can be continuous through streaming sources (i.e., APIs or sensors) or through one-time uploads from external sources. This data can be structured (databases, logs, financial data), semi-structured (HTML, XML, RDF, CSV), or unstructured (images, PDFs, Word documents). Raw data cannot be copied or exported out of SDC.
Curated
Data curation is the organization and integration of raw data collected from various sources. The curated data is annotated, so that the value of the data is maintained and made available for reuse and preservation. During the curation process, data is transformed from unstructured and semi-structured formats to structured formats; and data deduplication, obfuscation, and cleansing processes are conducted - resulting in high-quality data that enables researchers to elicit meaningful insights.
Published
Researchers create published datasets to disclose their research and allow other users to verify and reuse the data beyond their original purpose. Published datasets are a result of combining analyses on curated datasets in the SDC platform with other datasets or algorithms owned or created by a researcher or data scientist.
What's Next?
As a data analyst planning to do analysis in the SDC, use the steps below to get started.
Download the access request form , fill out the required details, and send an email to sdc-support@dot.gov. Once approved, we will send you an email with the instructions for accessing the platform.
Follow the instructions in the Welcome Email from the SDC. Review the Research Analyst User Guide .
Our Enablement Services team offers custom upgrades to help your project team along the way
Last updated October 2021