Stepping up the attack on the Congress, Prime Minister Narendra Modi on Wednesday accused the party of ignoring BR Ambedkar's contribution to the country and always giving the credit to one family.