NVIDIA NCP-AII Practice Test 80
Question 790
During an NVIDIA AI factory acceptance test, engineers discover: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability. Which TWO complementary elements are needed: the correct technical action and the strongest independent diagnostic evidence? Select TWO answers.
- Validate intended inter-node RDMA devices, transport selection and network reachability with multi-node NCCL tests.
- Test NCCL throughput and latency across representative message sizes and node counts with repeatable conditions.
- Inspect long-run NCCL logs, error rate, interconnect counters, rank liveness and timing variation.
- Inspect RDMA device state, NCCL transport logs, port counters, rank mapping and cross-node bandwidth.
- Assume every successful intra-node GPU benchmark proves cross-rack InfiniBand bandwidth is ready.
Explanation
Correct Answers
A:This is an appropriate correction for Inter-node NCCL east-west RDMA readiness: Validate intended inter-node RDMA devices, transport selection and network reachability with multi-node NCCL tests. Observed scenario: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability. The governing mechanism is: Local NVLink health does not establish end-to-end east-west cluster communication across NICs and switches.
D:This is the direct operational evidence to investigate: Inspect RDMA device state, NCCL transport logs, port counters, rank mapping and cross-node bandwidth. Observed scenario: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability.
Incorrect Answers
B:This alternative is relevant to NCCL bandwidth and latency payload sweep but does not directly resolve the current Inter-node NCCL east-west RDMA readiness issue: Test NCCL throughput and latency across representative message sizes and node counts with repeatable conditions. Observed scenario: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability.
C:This alternative is relevant to Multi-node NCCL repeated burn-in but does not directly resolve the current Inter-node NCCL east-west RDMA readiness issue: Inspect long-run NCCL logs, error rate, interconnect counters, rank liveness and timing variation. Observed scenario: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability.
E:This tempting but unsafe shortcut is not a sufficient remediation for Inter-node NCCL east-west RDMA readiness: Assume every successful intra-node GPU benchmark proves cross-rack InfiniBand bandwidth is ready. An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability. Local NVLink health does not establish end-to-end east-west cluster communication across NICs and switches.
Question 791
A production GPU cluster review identifies this condition: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability. Which diagnostic evidence should be obtained FIRST to substantiate the likely fault or configuration gap? Select ONE best answer.
- Assume every successful intra-node GPU benchmark proves cross-rack InfiniBand bandwidth is ready.
- Inspect NCCL debug traces, selected network plugin, environment variables and host network topology.
- Inspect RDMA device state, NCCL transport logs, port counters, rank mapping and cross-node bandwidth.
- Inspect nccl-tests latency, algorithmic and bus bandwidth, iteration variance and message-size curves.
Explanation
Correct Answer
C:This is the direct operational evidence to investigate: Inspect RDMA device state, NCCL transport logs, port counters, rank mapping and cross-node bandwidth. Observed scenario: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability. The governing mechanism is: Local NVLink health does not establish end-to-end east-west cluster communication across NICs and switches.
Incorrect Answers
A:This tempting but unsafe shortcut is not a sufficient remediation for Inter-node NCCL east-west RDMA readiness: Assume every successful intra-node GPU benchmark proves cross-rack InfiniBand bandwidth is ready. Observed scenario: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability. The governing mechanism is: Local NVLink health does not establish end-to-end east-west cluster communication across NICs and switches. The proportionate action is: Validate intended inter-node RDMA devices, transport selection and network reachability with multi-node NCCL tests.
B:This alternative is relevant to NCCL transport environment and benchmark reproducibility but does not directly resolve the current Inter-node NCCL east-west RDMA readiness issue: Inspect NCCL debug traces, selected network plugin, environment variables and host network topology. Observed scenario: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability.
D:This alternative is relevant to NCCL bandwidth and latency payload sweep but does not directly resolve the current Inter-node NCCL east-west RDMA readiness issue: Inspect nccl-tests latency, algorithmic and bus bandwidth, iteration variance and message-size curves. Observed scenario: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability.
Question 792
In a professional AI infrastructure deployment, a technician observes: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability. After corrective work, which observable result BEST confirms that the requirement is met? Select ONE best answer.
- A representative multi-node collective completes correctly using the approved high-speed network path.
- Benchmark runs can be repeated with the same approved transport and explainable performance changes.
- Assume every successful intra-node GPU benchmark proves cross-rack InfiniBand bandwidth is ready.
- Observed performance meets both small-message latency and large-message throughput objectives for intended jobs.
Explanation
Correct Answer
A:This is an observable verified result, not merely a claimed configuration change: A representative multi-node collective completes correctly using the approved high-speed network path. Observed scenario: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability.
Incorrect Answers
B:This alternative is relevant to NCCL transport environment and benchmark reproducibility but does not directly resolve the current Inter-node NCCL east-west RDMA readiness issue: Benchmark runs can be repeated with the same approved transport and explainable performance changes. Observed scenario: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability.
C:This tempting but unsafe shortcut is not a sufficient remediation for Inter-node NCCL east-west RDMA readiness: Assume every successful intra-node GPU benchmark proves cross-rack InfiniBand bandwidth is ready. Observed scenario: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability. The governing mechanism is: Local NVLink health does not establish end-to-end east-west cluster communication across NICs and switches. The proportionate action is: Validate intended inter-node RDMA devices, transport selection and network reachability with multi-node NCCL tests. A direct check is: Inspect RDMA device state, NCCL transport logs, port counters, rank mapping and cross-node bandwidth. The acceptance condition is: A representative multi-node collective completes correctly using the approved high-speed network path.
D:This alternative is relevant to NCCL bandwidth and latency payload sweep but does not directly resolve the current Inter-node NCCL east-west RDMA readiness issue: Observed performance meets both small-message latency and large-message throughput objectives for intended jobs. Observed scenario: An all-reduce job works between two hosts but fails when the full rack starts due to missing fabric reachability.
Question 793
The cluster operations team needs to evaluate this situation: A distributed workload is fast with megabyte messages but slow for small synchronization collectives. Which operational response is MOST appropriate to address the root issue? Select ONE best answer.
- Test NCCL throughput and latency across representative message sizes and node counts with repeatable conditions.
- Measure representative end-to-end dataset and checkpoint transfer rates from intended workers at target concurrency.
- Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output.
- Benchmark supported collective strategies and rank mappings against actual topology and message sizes.
Explanation
Correct Answer
A:This is an appropriate correction for NCCL bandwidth and latency payload sweep: Test NCCL throughput and latency across representative message sizes and node counts with repeatable conditions. Observed scenario: A distributed workload is fast with megabyte messages but slow for small synchronization collectives.
Incorrect Answers
B:This alternative is relevant to Storage streaming bandwidth under GPU load but does not directly resolve the current NCCL bandwidth and latency payload sweep issue: Measure representative end-to-end dataset and checkpoint transfer rates from intended workers at target concurrency. Observed scenario: A distributed workload is fast with megabyte messages but slow for small synchronization collectives.
C:This tempting but unsafe shortcut is not a sufficient remediation for NCCL bandwidth and latency payload sweep: Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output.
D:That option concerns NCCL algorithm choice and topology mapping. Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete.
Question 794
At a multi-node AI infrastructure handoff, the team records: A distributed workload is fast with megabyte messages but slow for small synchronization collectives. Which underlying infrastructure principle BEST explains the issue? Select ONE best answer.
- Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete.
- Collectives require coordinated communicator membership and consistent rank participation; partial launches can hang.
- Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output.
- RDMA verbs benchmarks reveal transport behavior not fully measured by ordinary TCP connectivity tools.
Explanation
Correct Answer
A:This is the governing infrastructure principle: Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete. Observed scenario: A distributed workload is fast with megabyte messages but slow for small synchronization collectives.
Incorrect Answers
B:This alternative is relevant to NCCL communicator and rank-to-device consistency but does not directly resolve the current NCCL bandwidth and latency payload sweep issue: Collectives require coordinated communicator membership and consistent rank participation; partial launches can hang. Observed scenario: A distributed workload is fast with megabyte messages but slow for small synchronization collectives.
C:This tempting but unsafe shortcut is not a sufficient remediation for NCCL bandwidth and latency payload sweep: Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output. Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete.
D:That option concerns InfiniBand verbs microbenchmark bandwidth latency. The proportionate action is: Test NCCL throughput and latency across representative message sizes and node counts with repeatable conditions.
Question 795
During an NVIDIA AI factory acceptance test, engineers discover: A distributed workload is fast with megabyte messages but slow for small synchronization collectives. Which diagnostic evidence should be obtained FIRST to substantiate the likely fault or configuration gap? Select ONE best answer.
- Inspect process lists, world-size and rank variables, GPU device assignments and NCCL initialization logs.
- Inspect nccl-tests latency, algorithmic and bus bandwidth, iteration variance and message-size curves.
- Inspect ib_write_bw or ib_write_lat output, MTU, negotiated link speed and HCA or switch counters.
- Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output.
Explanation
Correct Answer
B:This is the direct operational evidence to investigate: Inspect nccl-tests latency, algorithmic and bus bandwidth, iteration variance and message-size curves. Observed scenario: A distributed workload is fast with megabyte messages but slow for small synchronization collectives.
Incorrect Answers
A:This alternative is relevant to NCCL communicator and rank-to-device consistency but does not directly resolve the current NCCL bandwidth and latency payload sweep issue: Inspect process lists, world-size and rank variables, GPU device assignments and NCCL initialization logs. Observed scenario: A distributed workload is fast with megabyte messages but slow for small synchronization collectives.
C:That option concerns InfiniBand verbs microbenchmark bandwidth latency. Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete.
D:This tempting but unsafe shortcut is not a sufficient remediation for NCCL bandwidth and latency payload sweep: Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output. A distributed workload is fast with megabyte messages but slow for small synchronization collectives.
Question 796
A production GPU cluster review identifies this condition: A distributed workload is fast with megabyte messages but slow for small synchronization collectives. After corrective work, which observable result BEST confirms that the requirement is met? Select ONE best answer.
- Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output.
- All planned ranks join the intended communicator and finish the same collective without duplicate device placement.
- Observed performance meets both small-message latency and large-message throughput objectives for intended jobs.
- The collective returns numerically valid outputs across intended ranks at accepted latency and bandwidth.
Explanation
Correct Answer
C:This is an observable verified result, not merely a claimed configuration change: Observed performance meets both small-message latency and large-message throughput objectives for intended jobs. Observed scenario: A distributed workload is fast with megabyte messages but slow for small synchronization collectives.
Incorrect Answers
A:This tempting but unsafe shortcut is not a sufficient remediation for NCCL bandwidth and latency payload sweep: Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output. A distributed workload is fast with megabyte messages but slow for small synchronization collectives. Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete.
B:This alternative is relevant to NCCL communicator and rank-to-device consistency but does not directly resolve the current NCCL bandwidth and latency payload sweep issue: All planned ranks join the intended communicator and finish the same collective without duplicate device placement. Observed scenario: A distributed workload is fast with megabyte messages but slow for small synchronization collectives.
D:This alternative is relevant to NCCL device buffers and data integrity checks but does not directly resolve the current NCCL bandwidth and latency payload sweep issue: The collective returns numerically valid outputs across intended ranks at accepted latency and bandwidth. Observed scenario: A distributed workload is fast with megabyte messages but slow for small synchronization collectives.
Question 797
In a professional AI infrastructure deployment, a technician observes: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations. Which operational response is MOST appropriate to address the root issue? Select ONE best answer.
- Validate collective correctness over representative sizes and iteration counts alongside throughput.
- Test NCCL throughput and latency across representative message sizes and node counts with repeatable conditions.
- Run repeatable multi-node NCCL stress over representative durations, message sizes and full-rack participation.
- Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output.
Explanation
Correct Answer
B:This is an appropriate correction for NCCL bandwidth and latency payload sweep: Test NCCL throughput and latency across representative message sizes and node counts with repeatable conditions. Observed scenario: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations.
Incorrect Answers
A:This alternative is relevant to NCCL device buffers and data integrity checks but does not directly resolve the current NCCL bandwidth and latency payload sweep issue: Validate collective correctness over representative sizes and iteration counts alongside throughput. Observed scenario: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations.
C:This alternative is relevant to Multi-node NCCL repeated burn-in but does not directly resolve the current NCCL bandwidth and latency payload sweep issue: Run repeatable multi-node NCCL stress over representative durations, message sizes and full-rack participation. Observed scenario: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations.
D:This tempting but unsafe shortcut is not a sufficient remediation for NCCL bandwidth and latency payload sweep: Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output. An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations.
Question 798
The cluster operations team needs to evaluate this situation: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations. Which underlying infrastructure principle BEST explains the issue? Select ONE best answer.
- Collective performance is not useful when data integrity or communicator completion fails.
- Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete.
- Collective scheduling and transport are sensitive to communicator geometry and payload sizes; one algorithm is not universally optimal.
- Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output.
Explanation
Correct Answer
B:This is the governing infrastructure principle: Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete. Observed scenario: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations.
Incorrect Answers
A:That option concerns NCCL device buffers and data integrity checks. The proportionate action is: Test NCCL throughput and latency across representative message sizes and node counts with repeatable conditions.
C:That option concerns NCCL algorithm choice and topology mapping. The proportionate action is: Test NCCL throughput and latency across representative message sizes and node counts with repeatable conditions.
D:This tempting but unsafe shortcut is not a sufficient remediation for NCCL bandwidth and latency payload sweep: Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output. An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations. Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete.
Question 799
At a multi-node AI infrastructure handoff, the team records: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations. Which diagnostic evidence should be obtained FIRST to substantiate the likely fault or configuration gap? Select ONE best answer.
- Inspect NCCL debug traces, selected network plugin, environment variables and host network topology.
- Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output.
- Inspect local device topology, nccl-tests rank count, all-reduce bandwidth and link error metrics.
- Inspect nccl-tests latency, algorithmic and bus bandwidth, iteration variance and message-size curves.
Explanation
Correct Answer
D:This is the direct operational evidence to investigate: Inspect nccl-tests latency, algorithmic and bus bandwidth, iteration variance and message-size curves. Observed scenario: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations.
Incorrect Answers
A:That option concerns NCCL transport environment and benchmark reproducibility. Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete.
B:This tempting but unsafe shortcut is not a sufficient remediation for NCCL bandwidth and latency payload sweep: Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output. Observed scenario: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations. The governing mechanism is: Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete. The proportionate action is: Test NCCL throughput and latency across representative message sizes and node counts with repeatable conditions.
C:That option concerns Intra-node NCCL all-reduce test scope. Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete.
Question 800
During an NVIDIA AI factory acceptance test, engineers discover: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations. Which THREE elements together establish the appropriate corrective action, independent test evidence and verified successful outcome? Select THREE answers.
- Inspect process lists, world-size and rank variables, GPU device assignments and NCCL initialization logs.
- Inspect nccl-tests latency, algorithmic and bus bandwidth, iteration variance and message-size curves.
- Validate intended inter-node RDMA devices, transport selection and network reachability with multi-node NCCL tests.
- Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output.
- Observed performance meets both small-message latency and large-message throughput objectives for intended jobs.
- Test NCCL throughput and latency across representative message sizes and node counts with repeatable conditions.
Explanation
Correct Answers
B:This is the direct operational evidence to investigate: Inspect nccl-tests latency, algorithmic and bus bandwidth, iteration variance and message-size curves. Observed scenario: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations. The governing mechanism is: Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete.
E:This is an observable verified result, not merely a claimed configuration change: Observed performance meets both small-message latency and large-message throughput objectives for intended jobs. Observed scenario: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations.
F:This is an appropriate correction for NCCL bandwidth and latency payload sweep: Test NCCL throughput and latency across representative message sizes and node counts with repeatable conditions. Observed scenario: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations. The governing mechanism is: Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete.
Incorrect Answers
A:This alternative is relevant to NCCL communicator and rank-to-device consistency but does not directly resolve the current NCCL bandwidth and latency payload sweep issue: Inspect process lists, world-size and rank variables, GPU device assignments and NCCL initialization logs. Observed scenario: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations.
C:This alternative is relevant to Inter-node NCCL east-west RDMA readiness but does not directly resolve the current NCCL bandwidth and latency payload sweep issue: Validate intended inter-node RDMA devices, transport selection and network reachability with multi-node NCCL tests. Observed scenario: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations.
D:This tempting but unsafe shortcut is not a sufficient remediation for NCCL bandwidth and latency payload sweep: Declare a fabric performant solely from the single largest NCCL bus-bandwidth value in one test output. Observed scenario: An AI team reports only peak link throughput while ignoring tail latency in repeated all-reduce operations. The governing mechanism is: Network and collective performance vary by size, concurrency and serialization overhead; one peak number is incomplete. The proportionate action is: Test NCCL throughput and latency across representative message sizes and node counts with repeatable conditions. A direct check is: Inspect nccl-tests latency, algorithmic and bus bandwidth, iteration variance and message-size curves. The acceptance condition is: Observed performance meets both small-message latency and large-message throughput objectives for intended jobs.