As far as I know (I lived in Kaliningrad for 4-5 years), they were violently made to move in Germany in year 1947. A couple thousands were still there up to 1949, but only because they were some important specialists (their help was necessary to rebuild the city). These people couldn't get a soviet citizenship, which is why they were also sent to Germany later.
Last questioning in year 2010 has showed, that there are still 1676 Germans living there.
Source: wiki