NLBs push the connection directly to the endpoint that you are targeting. Thererfor you need to setup the security groups at the endpoint (dagit server) with the IP restrictions that you want to have in place.
NLB don't do any security, they are L3 load balancers and simply do reverse NAT and connection tracking.