Table 2.
SF-Relate scales efficiently to large data sets
| Data set | SF-Relate | All-pairwise | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Runtime | Communication | Runtime (estimated total) | Comm. (estimated total) | |||||||
| Step 1 | Step 2 (MHE) | Total | Step 1 | Step 2 (MHE) | Total | |||||
| Phase 1 | Phase 2 | Phase 1 | Phase 2 | |||||||
| UKB-200K | 1.8 min | 14.0 h | 0.5 h | 14.5 h | — | 46.6 TB | 0.5 GB | 46.6 TB | 1.3 years | 32.5 PB |
| UKB-100K | 49.5 sec | 7.05 h | 0.23 h | 7.29 h | — | 23.85 TB | 241.7 MB | 23.85 TB | 112 days | 9.8 PB |
| AoU-20K | 18.6 sec | 5.65 h | 0.11 h | 5.79 h | — | 6.2 TB | 77.6 MB | 6.2 TB | 18.8 days | 2.31 PB |
-
We report the runtime and communication costs for individual steps of SF-Relate described in Methods. The runtime and communication costs for setting up the cryptographic keys are 40.4 sec and 1.7 GB, respectively, constant across all experiments. We also show the estimated total costs of running all-pairwise comparisons and determining the closest relationship for each individual using MHE.
